While there are some good sites that host indexes that you can search, there really was nothing that met her needs to compiling and coalescing the information and for capturing and presenting her conclusions. The amateur software was obviously not sufficient for what she needed and there seemed to be not good data format for any of the data except for pedigrees, which actually is the result of research, ie the conclusion, not the pieces of data that lead up to determining a pedigree.

I told my aunt that I imagined that genealogy data is pretty messy, that it’s often incomplete, inconsistent, contradictory, in different levels of detail, quality, uncertainty, and authority. And rather trying to fit messy data into defined buckets or categories or pedigrees, what there really needs to be is something that embraces the messiness and just captures everything and what the system produces is probable profile of an individual. Individuals and families become more of a statistical probability of a convergence of loose data points, rather that a “Person” in the system. This may sound a little harsh and impersonal, but we often don’t have data sufficient to make an authoritative call on what a “Person” is based on the data available. So we have to switch to thinking of the data as revealing information about an individual with a non-definite amount of certainty.

My aunt of course lit up at this, and after I said I have experience in a system that can do just that, I was invested. I bought several books on genealogy for professionals and my understanding of them and the field increased tremendously. A professional genealogist is part historian, part biographer, part information analyst, part linguist, part author, and forensic expert, and other things too I’m sure. Evidence is paramount. If you can’t provide sources and correct citations for the information and conclusions you’ve drawn, then you have no conclusions that anyone can accept. The most common publishing avenue is in document form, as in reports, book, and articles. The sharing of the data itself is not where it needs to be, often being trapped in footnotes, in-document tables and charts, and of course whiteboards (or photos of whiteboards).

The RootsTech Developer Challenge

The RootsTech Developer Challenge was a good opportunity for me to make reality ideas I have had on how to address the need for improved genealogy systems. The challenge is billed more for “apps” to enhance the public’s engagement with genealogy and to foster increased interest in the field. I did not think the problem or next step is better apps. I think the problem is how the data is handled. Even amazing apps aren’t going to be able to compensate for issues with a weak system for handling the data. What I thought is that we need a fundamentally different way to handle genealogy data and embrace the messiness.

What we needed was a system that is designed to take in data on just about anything, from individuals to locations, from factoids of an individual’s life to full and complete pedigrees. We needed a system that did not seek for the one correct data set on a person but rather a high tolerance of uncertainty, contradiction, and even error. Some genealogy systems in the past sought to create a single pedigree for everyone but what ended up is people overwriting other’s previous entries and errors that were introduced were perpetuated, causing a gradual degradation of the quality of the data in general in the system. We needed a system that could take in anything and with quality and certainty measures be able to tag the data points so users could know the quality of the data. And we need the users themselves to be able to set some of the quality and certainty measures.

We also need users to be able to upload the results of their work, research and conclusions, not just in report or final form, but even the fragments and bits that they have. Users need to be able to make these data public, keep them private, or share them with others that they choose. Users need to be able to incorporate these data in their searches, and they need to be able to pick and choose what kind of data (public, private or shared) when they search. They also need to be able to choose the quality and certainty level of data they want to include in their search to improve the signal to noise ratio of their results.

And why can’t the system go through the data and infer some results? If John Smith was divorced from Alice in 1834, why can’t the system infer that John Smith is male, was married before 1834, and that his wife was probably named Alice Smith? And why can’t the system infer that the John Smith in that divorce record is the same John Smith in another census record that says he lived in Pleasantdale, Maine and had two kids, or at least provide a level of probability? Why can’t then the information between these two records be combined when I search for “Smith divorced Maine” and have a hit, showing me all the combined information for person with the quality and certainty levels displayed?

Why can’t users add their conclusions, assertions, and inferences, too? Why can’t Sally add that she found an obituary for John Smith in Pleasantdale in an old microfiched copy of a newspaper and it says he was survived by three children? And after all these derivative data points are added in the system, why can’t I trace each one of them back to their original sources, with contact information of the people who captured or entered the information?

It might sound nice but pie-in-the-sky, but that’s what I did and that’s what I entered into the RootsTech Developer Challenge. Granted the application is ugly (think Geocities…beta) and I’m sure it has bugs, but everything I described above is included and it works and works fast (all subsecond with a 4GB database), with the exception of being able to include certainty levels in searches. I don’t expect it to win and I am kind of done with that challenge because I feel like I figured it out and now I want to move on with it, but I enjoyed the challenge.

How it Works

The data model was the single most important factor of getting this system to work. Relational data models with tables and columns is just insufficient because the data could be virtually anything and could be virtually in any structure, and that just doesn’t fit in tables and columns. XML is better because it can allow any structure in a document with an field and any values, but that still doesn’t make searching the data any easier. To search the data and make sense of the values you need to know ahead of time what fields are available. A Semantic Data Model provides a way for any data to be of any type and associated with any other thing, be it a value or an entity, so that’s the model I chose.

The system I built at its heart is a Semantic Data system with some modifications. Semantic data is in Subject – Predicate – Object form. The subject is anything, often it is an id or key of some “thing” but it can be anything. The Predicate is the “type” of relationship between the Subject and the Object. A Predicate can be for a value (“Eye color is”) or it can be for the relationship between two things (“is Father of”). The Object is either an value (“Blue”) or the id or key of some entity (“123-John-Smith”). The Predicate usually tells you if the Object is a value or another entity, which would probably be used elsewhere as a Subject in some other fact.

This kind of triple, S-P-O, is called a tuple and can be used to model virtually any kind of data. For genealogy, there may be a tuple like:

“123456789″ “first-name” “John”

and another tuple

“123456789″ “last-name” “Smith”

and another

“123456789″ “married-to” “4443333222″

and another

“4443333222″ “birthplace” “Chicago”

and another

“11111111″ “father-of” “4443333222″

and another

“11111111″ last-name” “Grant”

etc.

So in the above example, we know that there is someone with a name of “John Smith” who married someone who was born in Chicago and whose father’s last name was Grant. Even without complete information, we can “walk the graph” and go from person to person using relationships defined in the data and view the data we have for each person and at each step. Note too that the relationships can be bi-directional. We may know that A is the father of B or we may know that B is the daughter of A. Either way we can walk the relationship graph and infer the parent-child relationship even if that relationship is only original defined one direction. Better yet, the system can make the inference and add that inference as a new tuple into the system.

For a collaborative semantic genealogy data system, Subject – Predicate – Object is actually insufficient. Three other aspects need to be added: Time, Source, and Quality. If we know that in an 1850 census is says John Smith was 34, we want to capture that original data as-is with no interpretation in order to preserve the data integrity. So we would have:

“12345-page-4-line-2″ (or whatever we decide the subject should be) “Age is” “34″

But that fact was not always true, just when the census worker captured the information. So we need to add Time, which in this case is “1850″ or as much as we know. So now we have Subject – Predicate – Object – Time (S-P-O-C):

“12345-page-4-line-2″  “Age is” “34″ “1850″

But what if that data is coming from something written in someone’s Bible. We want to capture that too, So now we’ll have Subject – Predicate – Object – Time – Source (S-P-O-C-E):

“12345-page-4-line-2″  “Age is” “34″ “1850″ “Alison-Grant-Family-Bible”

Now the source would be a key to some other data that would have detailed information about the source and contact information for whoever got the information from the source. But how reliable is this data? Is it for the right John Smith? Is the information legible in the Bible? Is it firsthand, secondhand, or more? We need a quality value, so know we have Subject – Predicate – Object – Time – Source – Quality (S-P-O-C-E-Quality):

“12345-page-4-line-2″  “Age is” “34″ “1850″ “Alison-Grant-Family-Bible” “4″

Assuming a scale from 1 to 10 and 10 is absolutely certain. Now we have a model that can capture virtually any data, for anything, true at a particular time, of any quality, and noting the source. For sharing and privacy we can either add those into the tuples, or use system permission controls to secure the tuples themselves.

Yeah But How Can This Be Fast?

We covered conceptually how to model the data to achieve the goals I listed at the beginning. Fortunately there are several products in the industry for Semantic or Graph Databases. I chose to use MarkLogic because it was familiar to me, I knew it was extremely fast, can scale to the petabyte level, but most importantly it has very advanced language tools. Other graph database may be fast linking different pieces of data together, but I also needed one that was fast when searching for first names that sound like “John”, or contain “John”, or are spelled similar to “John.” I needed something that could find word stem hits, like when I search for “run” I get “ran”, “runs”, “running” also. For this free text searching (which is almost entirely against the Object in the tuple) I need a powerful search system. So MarkLogic gave me both the free text searching and the fast data linking, and security down to the user and tuple level, which can scale to petabytes. I suppose this system could be build on other platforms, but I’ll leave it to the reader as an exercise to prove out.

The implementation details and actually code used can be pretty complicate and lengthy, but for those interested almost all the magic is in the prolific use of cts:queries in MarkLogic.  For searches such as “Family of John Smith” there is a Predicate Resolution step which builds a list of all Predicate types that are part of a “family” type which is determined by rules (“husband-to”, “brother-of” etc). Then this Predicate Set is sent as exact values to match in the P. The queries against Objects (O) are stemmed, case-insensitive, diacritic-insensitive, punctuation-insensitive, and whitespace-insensitive. I also created a Double Metaphone custom index on all the values for Objects of Predicate types of names and locations. So I took all unique values of O for those types and created a file for each type of the calculated Double Metaphone value, which MarkLogic provides an API for. Then the Object Resolution step include the original value typed and all the values from the Double Metaphone index (which are existing values in O) which have the same Double Metaphone value and which are then also included in the O set to the query. All linking and joins are done through the Subject (S). For tuples that have been linked “Same-As”, I query all Objects, get their subjects and filter them to the ones that have Same-As matched within the result set. This is the Subject Resolution Step. And voilà, I have my results.

Relevancy is not so good yet, and all the sharing and access control I’ve done via yet another field to the tuple: Sharing. But I’m rethinking that and considering using document permissions instead. Below are a couple of functions so people can see some of the code. It is most of the code in a free text search field that expects that you are going to include a first name and last name in your search.

What’s Left?

I don’t think I’ve answered the need of what my aunt needs for her individual research, but this provides the engine and capability to build off of. There lots of application interface coding left to do, too. Getting data for this has actually been pretty hard. Any genealogy data dumps I could get were on CDs from the 90s in Infobases format, with threatening EULAs about not using this data in another system. So I didn’t use that. Oddly there is a county in Maine that has a decent amount of census data online (trapped in HTML pages): Aroostock County. The State of Maine has several databases posted online (mostly in Access databases) like military record, divorces, court proceedings, and Revolutionary War Land Grants. I spent about half of my time getting the data and processing it into a format I could use, eventually in semantic form.

This system can be used as a way for researcher to enter share, and publish, their work and conclusions, and they can do so without any original data being in the system. They can make the Subjects individuals on the big genealogy sites, and the can make the sources point to those specific URLs elsewhere. This would then effectively be a big meta data system. The data need not be limited to name and date information, but can have photographs, scans, video, pdfs, and any other kind of data. But I really hope I can either get data or work with one of the big website to incorporate these capabilities. It’s just time and money.

declare function free-search($phrase, $sources, $start, $end) {

	let $terms := fn:tokenize($phrase, " ")[. != ""]
	let $subjects := reduce((), $terms)[$start to $end]
	let $subject := get-all-same-as-subjects($subjects)
	let $subjects :=
		cts:search(/t,
			cts:and-query((
				cts:element-value-query(xs:QName("s"), $subjects, "exact")
			,
				get-cts-search-source-query($sources)
			))
		)/s/text()
	let $subjects := fn:distinct-values($subjects)

	return
		<results>
			{
			for $s in $subjects
			let $family-name-hit := (/t[s = $s][p = $type:person-family-name]/o/text())[1]
			let $given-name-hit := (/t[s = $s][p = $type:person-given-name]/o/text())[1]
			let $name := fn:concat($given-name-hit, " ", $family-name-hit)
			let $query :=
				cts:and-query((
					cts:element-value-query(xs:QName("s"), $subjects, "exact")
				,
					cts:element-word-query(xs:QName("o"), $terms)
				,
					get-cts-search-source-query($sources)
				))

			let $t := cts:search(/t, $query)

			return
				<hit>
					<s>{$s}</s>
					<name>{$name}</name>
					{
					for $at in $t
					return
						<highlight>
							{$at/p}
							{$at/o}
							{$at/c}
							{$at/e}
						</highlight>
					}
				</hit>
			}
		</results>
};

declare function reduce($subjects, $terms) {
  let $matched-subjects :=
    if (fn:empty($subjects))
    then
      cts:search(/t,
        cts:element-word-query(xs:QName("o"), $terms[1])
      )/s/text()
    else
      cts:search(/t,
        cts:and-query((
          cts:element-value-query(xs:QName("s"), $subjects, "exact")
        ,
          cts:element-word-query(xs:QName("o"), $terms[1])
        ))
      )/s/text()
   return
     if (fn:count($terms) = 1)
     then fn:distinct-values($matched-subjects)
     else reduce(fn:distinct-values($matched-subjects), $terms[2 to fn:last()])
};

declare function get-all-same-as-subjects($subject) {
get-same-as-subjects($subject, ())
};

declare function get-same-as-subjects($check-subjects, $found-subjects) {

let $other-subjects := fn:distinct-values(
for $check-subject in $check-subjects
return
cts:search(/t,
cts:and-query((
cts:element-value-query(xs:QName("p"), $type:person-same-as, "exact")
,
cts:or-query((
cts:element-value-query(xs:QName("o"), $check-subject, "exact")
,
cts:element-value-query(xs:QName("s"), $check-subject, "exact")
))
))
)/(s|o)/text()
)

let $new-subjects :=
for $other-subject in $other-subjects
return
if ($other-subject = $found-subjects)
then ()
else $other-subject

return
if ($new-subjects)
then get-same-as-subjects($new-subjects, ($new-subjects,$found-subjects))
else $found-subjects

};

I usually create an initial resource bundle of some sort in English and then use Google Translate to translate the strings in the bundle into various target languages. I automate this so that I just push a button or have the script execute in a trigger so that when I change or add to the resource bundle in English I can re-translate it easily.

Here is the type of script I use to make the call to Google Translate:

declare function local:get-google-translation($text, $source-lang, $target-lang) {

  let $url := fn:concat("http://translate.google.com/translate_a/t?client=t&text=", xdmp:url-encode($text), "&hl=", $source-lang, "&tl=", $target-lang, "&multires=1&sc=1")

  let $response := xdmp:http-get($url,
    <options xmlns="xdmp:http-get">
      <format xmlns="xdmp:document-get">text</format>
    </options>
  )

  return fn:tokenize($response[2], '"')[2]
};

let $text := "Rome (CNN) -- Transcripts published Tuesday capture the dramatic conversations between port officials and a cruise ship captain, who a judge ruled can be held under house arrest while Italian authorities investigate his role in last week's disaster."
let $source-lang := "en"
let $target-lang := "es"

return local:get-google-translation($text, $source-lang, $target-lang)


=> ROMA (CNN) - Las transcripciones publicadas el martes la captura de las conversaciones entre los funcionarios del puerto espectacular y un capitán de barco de crucero , que puede ser un juez dictaminó bajo arresto domiciliario mientras las autoridades italianas investigar su papel en el desastre de la semana pasada .

Node was created to provide an event-based web server programming model that better utilizes threads on the server, particularly when it comes to IO operations (like filesystem reads or database calls). So rather than a thread having to wait for an IO operation to finish before program execution continues, the thread implements a callback to be called when the IO operation is finished. This way the threads are able to server more request because they aren’t waiting for expensive operations to complete.

Who has this problem? Whose web application performance bottleneck is that their threads are waiting for IO to complete? If this is your problem, then I don’t think you’ve got a very good web application implementation. Let me explain why.

HTML Generation is usually not the slowest part

Given that a web application is correct, available, and secure, users care most about speed. They don’t care about your hardware utilization or how many requests are handled per server, and they also don’t care how fast your web app is on average, they care about how fast it is for them. Page load time and time-until-usable are what users are concerned about.

Looking at cnn.com, there are 89 requests, totalling a little less than 1MB, which took 4.24s for it to load for me. Of those 89 requests, 3 were HTML requests from cnn.com (1 for the HTML page and 2 for weather). The HTML from cnn.com is about 30KB…out of 1MB! So if you want to speed up your site, where is the best place to focus? HTML from the server that makes up 3% of the total weight, 3% of the total number of  requests, and 9% of the total page load time? Or would you focus on reducing the number of requests, the size of assets, and the caching of those assets?

Cnn.com’s HTML took 336 ms to get to me. Let’s say you made that 10x faster. You would have then reduced the total page time by 300 ms or about 7% of the total page load time and still get about 4 seconds for page load. You could have a 1 ms HTML response time and still have a slow site. The HTML generation and return time is usually not where the problem is for web application performance.

Most of the assets on a web page are static (meaning they don’t change per request) so they can be served by a cache server (so the origin server isn’t hit) and by the browser (so not even the cache server is hit). The origin server can generate the HTML and server up the static assets if needed, but it really shouldn’t do that very often because the browser cache and cache servers should be serving them. So then what you really need is a content server that is geared toward HTML generation, whether it be static or dynamic. So you have the origin server generating dynamic but cacheable HTML (like templated by little-changing info pages), and for handling dynamic but non-cacheable HTML (like search).

The content server should not need to do hardly any IO. Why would an HTML content server need to write to the filesystem? Even if it does, why does the web visitor need to wait on the result of that file write operation before seeing the server response? If you really need to write to the filesystem, spawn a thread or offload that operation to something else that can queue up write operations. Your content server doesn’t need to do it; it just need to invoke something else to do it.

If your content server is serving up dynamic content, what else can it be doing before it gets the data from the database? It’s primarily going to be formatting and creating presentation using the data from the DB, and if it has something it can be doing in the meantime I’m arguing it should be doing it. Something else can communicate with other services and cache HTML fragments or whatever. All the content server does is process content, so if it has to wait for the data, it waits.

But why would there be any IO for data that takes much time at all? If the data is so far removed from the presentation engine (the content server) that it blocks for any noticeable amount of time, you got a problem with data retrieval. The answer isn’t to create a callback for when the data finally arrives from the DB, the answer is to fix the problem of data coming back so slow from the DB.

Functional programming facilitates optimized and parallelized execution

One of the reasons I like functional programming is because the execution engine is able to parallelize function calls because functions only operate on data coming into the function and only output a result. Function don’t change properties or state on objects in memory. Since there’s no shared state or objects that can be accessed by two different processes, all operations are threadsafe. Better yet, with lazy evaluations like what MarkLogic does for many things, you can capture the result of a function call in a variable, but the execution engine doesn’t need to actually make the function call until you access something on that variable, which could be at any later point in  your program. In fact, if you never access the variable the execution engine may never actually call the function that returns the value for that variable. Order of execution becomes much less important because the functions have no side effects and can be executed whenever the execution engine decides. The execute of one function does not affect another, so you can execute them all at once, or whenever resources are available. With Node, you’d be writing code to do all that: optimizing the method calling yourself. Instead, use a functional language and let the execution engine do it for you.

The problem I have is processing a lot of data quickly

I have megabytes and gigabytes of data to query and format for display on a web page. I need to be able to find a needle in a haystack and transform it into presentation quickly, for every request. First I need to get the speed down for just one user because that is as fast as I can go (unless another user were to cache it). Then I need that speed to remain fairly constant at scale, both with web traffic and amount of content. I am less concerned about how many requests each server can handle because I can scale horizontally if needed for both traffic and content size. With MarkLogic I have extremely fast access to the content I need. There’s no IO blocking to speak of. Even if there were, the execution engine will do some optimizing so parts of my code can execute in parallel. I spend time reducing query times, not coding callbacks for them.

Node enthusiasts are front-end coders not wanting to do server coding

I have used Javascript for over 15 years. I learned it before I learned Java. It’s really not too bad. I think what has happened in the web developer community is that some people who know front-end programming have gotten all excited that they can use their front-end skills to program the server. In fact, they think that they can even move a lot of processing that used to be done on the server up into the browser, using the programming languages and techniques they are used to, and all of the sudden it’s revolutionary and cutting edge. That’s a big reason CouchDB gained popularity, because there was no need for server programming. With HTML5, some have the idea that we don’t even hardly need a backend service at all, just to persist some state once in awhile.

So the Node community has tried to sell Node as solving a fundamental problem with server programming (blocking IO calls) but that’s really not the problem with web page speed or even server speeds, especially per request. I think the real reason is that they are mostly novices that want to use Javascript for the server side but they use the “blocking” argument to convince others. All the Node enthusiasts I know, some personally, are not very skilled server programmers but have pretty strong front-end skills. This revolution is more about front end coders not having to deal with the server side than any breakthroughs about how to do the server side. And the exuberance and arrogance from enthusiasts is meant to shame non-enthusiasts into thinking they’re old school, antiquated, or unable to learn new things, that this is the future and in a few years we’ll all be programming in Javascript and if you don’t get on board you’ll be out of a job (I heard this first-hand). Node has to be adopted, otherwise all these front-end coders will have to learn server programming.

But there are lots of things I like about Node, but not the community. I plan on using Node for easy HTTP server programming and for handling a large number of connections. But I need a Big Data server and a content server to generate dynamic and personalized HTML and to handle search. I’ll offload the HTML assets and cache as much as possible to cache servers, and I’ll optimize the front-end code to increase performance. Blocking calls, including IO, are just not one of my problems.

Global Namespaces can be added at the Group or Application level in MarkLogic. Through the Admin Interface on port 8001 or through the API you can set a prefix and namespace URI which is accessible in the code. You set it on the Group and then all App Servers in the Group will be able to access it, or you can set it on the App Server. The App Server’s namespace will override an existing Group namespace.

So if I wanted to set the type of lane my code is running in, I could set a namespace at the Group level that has a prefix of “lane” and a URI of “prod”. The following code would get the value:

fn:namespace-uri-for-prefix("lane", <lane:blah/>)
=> prod

And if I wanted to set some endpoint, I could create a namespace on the Group with the name “endpoint” and URI “http://mysite:6005″

fn:namespace-uri-for-prefix("endpoint", <endpoint:blah/>)
=> http://mysite:6005

Since these are global namespaces you don’t have to declare the namespace in the prolog, so you don’t need any more code than shown above.

Granted this is not using global namespaces for their intended purpose, but it seem to work pretty well.

Better binary content handling

MarkLogic has always been able to store binary files in the database, but if the files were too big or if you had too many files, your caches may have been adversely affected and your database merges may have taken longer than they needed to. In the past, when we had a lot of binary content that we wanted to serve off of a MarkLogic-powered website we would keep the binary files on the files system and just put the metadata file in the MarkLogic database. This worked fine, even streaming the files off the filesystem through MarkLogic, but we had to code the implementation and we always had to make sure the metadata files were in sync with the binary files on the filesystem. We don’t have to do this anymore with MarkLogic 5.

MarkLogic 5 introduces Rich Media Support which means that large binary files are handled differently than XML and text files under the covers in the server. There is a configurable threshold for the size of a binary file to be considered “large” as to be handled in a more efficient way. These large binary files are handled by MarkLogic as efficiently as if you saved them to the filesystem yourself. But you don’t need to use an special API or different functions that you would use for the XML and text files. You just insert the file using xdmp:document-insert() and MarkLogic will handle the rest.

Configuration importing and exporting

The Administration Interface on port 8001 provides a nice graphical, point-and-click interface for managing and configuring a MarkLogic installation. But for mature implementations, you’ll probably want a way to declare the settings for the servers, database, forests, etc. and script the configuration changes. There are several good implementations that do this outside of MarkLogic, but now you can just export the settings of an installation and get the full configuration settings in a XML file. You can import this XML file into a separate machine to stand up an installation with the exact same settings. You can also check in the configuration settings file into source control, make changes to it, and re-import the file back into the MarkLogic installation to affect those changes. As part of troubleshooting you can take a fresh export of the settings of an installation and compare those settings to the configuration settings file you had in source control to see if there were any inadvertent changes to the installation.

Getting the original URL of the request

This may seem to be a minor feature but one that can save me code and complexity. It’s always been possible to get the request URL from within XQuery code by calling xdmp:get-request-url(). But this returns the URL after the URL rewriter has rewritten the URL. What if you wanted to get the URL before the URL was rewritten? In previous versions of MarkLogic you’d have to get the request URL (by calling xdmp:get-request-URL()) in the URL rewriter itself and adding the original URL as a parameter to the rewritten URL. For example,

fn:concat("/new/url?orig-url=", xdmp:get-request-url())

Then in subsequent code you’d get the original URL by getting the request field, like xdmp:get-request-field("orig-url"). That works but it can be a pain if you forget to add the URL as a parameter, or you make in error in the code to retrieve it. But now in MarkLogic 5 you can just call xdmp:get-original-url() which will return the URL as it was before the URL rewriter changed it. Less code I have to write. Less complexity. Fewer bugs.

MarkLogic is fast in terms of performance but also in terms of development time. I spent ten years in the Java world and time-to-market was extremely important, and is still is now. I have never been able to implement mature, high-performance, enterprise solutions faster on any other platform than on MarkLogic. The new features of MarkLogic 5 that excite me the most are the ones that reduce that time-to-market for me even more. Most if not all of these features are the results of customers lobbying for them, and MarkLogic has listened. I have been vocal about binary content handling and now it’s part of the server. I’m looking forward to this new version so I can continue to push the boundaries of delivering solutions for my customers in less time and with less risk.

Similarly, database guys have pointed out to me that XML is still “relational” in that there are still relationships among the data. So maybe a better descriptor is “tabular data” when referring to databases that use tables, rows, and columns.

So I have been using different terms once in awhile that seem to get the points across better. I refer to Oracle, MySQL, and SQL Server as databases that store data in a “tabular” form. And I refer to MarkLogic as storing “variably-structured data” which immediately begs for a definition which I say:

Variably structured data is data that doesn’t necessarily conform to a schema imposed on it by the database, although the data may have internal structure and that internal structure may be the same as other data in the system, or it may vary to a degree. The internal of the structure may vary over time as the data itself changes.

It seems like there’s really nothing “semi” structured about XML at all. It’s structured, but the data itself gets to define its own structure. That’s key point #1: the system doesn’t impose structure, rather the data does it on itself. Key point #2: that structure can vary across documents, and within a document over time. Hence, the structure is variable.

Maybe I’m going against some long-standing academic history here but I don’t like “semistructured” and I want to stop using it. What do you all think?

First, there are some guiding principles that I follow when it comes to xquery web apps on MarkLogic. They are not set in stone, and individual projects may deviate. But these are my starting points:

1. Embrace the functional programming paradigm and the XML data model. Don’t fight it. Don’t try to turn XQuery into Java, or XML into an object model. Don’t try to reproduce hibernate, spring, jsp, php, rails, or any other object-oriented framework in MarkLogic. Let the server do its thing. You bought the server, so use it for what it is. Otherwise, write your own server.
2. Use a conceptually simple data model. Don’t start off with creating a data model structured with how it will be queried. Think of the entities that you will need and keep the number of them low. Have each type of entity have minimal references to other entities and keep them coarse. Keep it simple.
3. Make the application data- and configuration-driven for flexibility. Write the application so that it works according to the data it has. Facets are an example of this in search. The facets that are displayed are based on what came back in the results. You don’t have to specify up front what the individual facets will be. The server will figure that out at runtime. Build navigation and other options off of values in the DB. Write modular code that is discoverable at runtime. This is an advanced topic that can’t be sufficiently covered here, but the Masquerade approach (discussed below) is example of this.
4. Leave the HTML files intact. A rich interface requires plenty of HTML work so leave it alone. Don’t shred the HTML into XQuery. Don’t make the HTML only work on through a server. Let the front-end development take place in a static context, and then make replacements as needed dynamically at runtime. When the HTML needs to be modifued (which is constantly) the front-end dev can easily do it because the file didn’t have to change to be served on the server. The Nonob approach discussed below is an example of this.
5. Let collections and permissions restrict the data access for the user. Don’t write your own access restriction. Write code so that it just operates on the correct data based on the user’s permissions and the current collection in use.
6. Code to “bare metal” as much as possible. Use the built-in APIs if they can do what you need, even if you have to compose them together a little. Avoid custom functions just to “keep the code clean”. Avoid creating “gateway” functions, that is, functions that get the data the “right” way for you. Let the code get the data it needs, when and how it needs it. Not only does this reduce the amount of code you write, it allows you do surgically get just the data you want using predicates you want. It also reduces bug risk because the built-in APIs are going to be much more likely to be bug-free than something custom you wrote.
7. Use static javascript and CSS assets. MarkLogic is a powerful markup machine so use it to handle all the markup, and let the js and CSS be static and cacheable. This way most requests to the server are just for new markup that will have very little jas or CSS in it.

Ok, so how exactly do you do this? Below are some approaches, patterns, libraries, and techniques that my team used on the application:

1. Non-obtrusive HTML replacement (nonob).In this approach, the front-end dev creates the static HTML with whatever js and CSS assets he wants, using dummy data in the HTML, so that it’s fully mature right off of the filesystem. He then checks the HTML and assets into source control and the XQuery dev then finds which parts on the HTML page need to be dynamically replaced and writes replacement rules in an XQuery file. The XQuery dev checks in his code and leaves the HTML alone. The MarkLogic server executes the XQuery file, pulling in the HTML, makes the replacements using the the rules in the file and pulling in whatever data it needs, and returns the resulting HTML to the browser. The assets are also in MarkLogic so those requests just simply return the requested js and CSS files that the front-end dev put it. This is probably the single biggest time saver in development. There are several reasons why:
   • Front-end development and server development can happen in parallel. Given a design, both the front-end dev and XQuery dev can determine what needs to be done and they can bot start working on their areas. It’s common for the front-end dev to check in HTML, which then the XQuery dev refers to for replacements, even while the front-end dev keeps making changes to that HTML.
   • Changes can be made at any time by front-end or XQuery devs. At any point, if either front-end devs or XQuery devs need to change code or HTML, they can do it without having to consult with each other. There are limits, but this is true for the most part.
   • XQuery code only needs to focus on making the replacements, not building the page. The only XQuery code to write for HTML pages is getting the necessary data to make the replacements. Let the front-end guy deal with doing all the HTML.
   • The same XQuery code can do replacements on different HTML files. You can have one HTML page for desktop, one for tablet, one for mobile, one for IE6, whatever, and the same XQuery code can be used to do the replacements for any of the files. You can also have multiple views (Large image, no image, banner, etc) of a page and the same XQuery code is used. It’s just that the input HTML page changes.
2. Masquerade.This is a technique that abstracts the addressing of a resource from the code that will service that resource. Friendly URLs are a common example of this where there are rules or configuration that map URLs to pages. Masquerade takes it a few steps farther by querying the DB for code that has indicated that it responds to that URL and the appropriate code is used to generate the HTML based on the session or user agent. Here is how it works:
   • Request comes into the rewriter in MarkLogic
   • The rewriter queries the DB (particular collection, and using the permission set of the user) to find a “masq” that has an “url” element that matches the URL path. An example XML file for a masq may look like this:

     <masq key="aboutus">
         <url type="preferred">/about-us</url>
         <url>/aboutus</url>
         <url>/about-us.html</url>
         <view type="mobile" xml:lang="en">
              <html>about-us.html</html>
             <code>about-us.xqy</code>
         </view>
         <view type="desktop" xml:lang="en">
               <html>about-us-mobile.html</html>
              <code>about-us.xqy</code>
          </view>
     </masq>

   • The rewriter picks the best “view” of the masq that fits the client.
   • The rewriter calls the code that the view specifies and passes the starting HTML that the view specifies
   • The code uses nonob to make replacements on the HTML and returns the HTML back to the browser.
   • Making a new page and new page rule involves writing the HTML, the code, and creating a masq XML file. No changes to the rewriter are needed because masqs are queried at runtime
   • Pages that are off-limits to the certain users can be protected by setting the permissions on the masq XML file so that the query for masqs would not find the restricted ones for that user. You can also set the permissions on the xqy file.
3. Nonob i18n.In order to facilitate the page being in multiple languages, the nonob i18n library does the following:
   • In the static HTML that the front-end developer writes, custom HTML attributes are added to all the part of the page that need to be i18n-ized. These custom attributes do not interfere with any HTML coding tools and browsers ignore attributes they don’t understand, but the attributes are removed before being sent to the browser at part of the nonob replacement.
   • Here’s an example of i18n attribute on a div. The value of the i18n attribute is the key to lookup the translation in a resource file.

     <div id="mydiv" i18n="myapp.welcome-page.title">Welcome to my site!</div>

   • Here’s an example where you want to i18n-ize an input field:

     <input type="textbox" i18n="pages.register.name.placeholder"
     i18n-target="value" value="Name"/>

   • The resource files are stored by language. And example of one might be:

     <resource-bundle xml:lang="spa">
         <resource key="pages.register.name.placeholder">Nombre</resource>
         <resource key="pages.register.submit">Submito</resource>
         ...
     </resource>

   • The resource to use is determined at runtime using the user’s language preferences sent in the HTTP headers (or previously set settings) and the resource bundle language available. So if the browser says, “ru,es,en” then the framework will pick the most appropriate language to use, which in our example is Spanish (es), because there is no Russian version.
   • After nonob has made the replacements (which is does in one pass through the HTML), and then i18n-nonob does it’s replacements in one pass, where it finds all the i18n attributes, gets the values from the resource files for the given language, makes the replacements, and returns the HTML to the browser.
   • This requires one line of code to use the library, and the only effort is to add the i18n attributes to the static HTML and create the resource files. Note that the page can dynamically set i18n attributes if it wants in the nonob step because the i18n-nonb step is just going to parse the whole HTML again after nonob is done.
   • Nonob + i18n nonob takes about 50 milliseconds usually
   • Nonob + masquerade + i18n-nonob provides a lot of flexibility and permutations with minimal coding.

Those are the more important parts that contributed the most to rapid development. We also already had a configure and deployment tool, and an authentication library so we didn’t have to spend much time with those. The book I am writing goes into better detail about how to do some of these things.

Enjoy

Isn’t it the same with data? When you define a table in a relational database, you are defining what data parts an entity can have. And when you put data in that table, you are mixing that entity’s data with all the other entities that have data in that table. You are essentially coupling the data among all the entities in that table. Now you can’t separate the entities very easily and you can’t change the data structure or types for an entity without affecting all the other entities.

In a document-centric model, if each entity has it’s own document you can insert, delete and modify them with no effect on other entities. You can also create new types of entities, and even convert some old entities into the new type, without affecting other entities.

But even if you are using XML you still may gravitate to data coupling if the data gets sharded too much. I use a rule of thumb that an entity is something that has it’s own lifecycle. It can reference other entities (like a foreign key) but those are references to other entities, not sharded data parts of the same entity.

For example, a Person entity may have a “name” element which does not exist independently and does not have its own lifecycle. So “name” is an element of a Person. But that Person may be in a Company and that Company does exist independently from any Person and it has it’s own lifecycle, so Company should be a separate entity and each Person entity would have an id reference to the Company that it belongs to.

Now what happens if a Company entity gets deleted when a Person references it? Then you have an ID to nothing, but the Person still exists, still can be viewed, modified, and potentially reassigned, but the Person entity is still complete. And when a Person referencing some Company is deleted, the Company doesn’t know, doesn’t care.

I typically expect that the number of data models for entities in a MarkLogic web application should be able to be counted on one hand. And usually starting with what makes conceptual sense is the best approach: what “entities” do there seem to be that have their own lifecycles and how do they relate to each other?

This reduces data coupling so changes can be made easily and with low risk. Entities can be inserted and deleted without affecting other entities. It also tends to foster queries that are simple and “fully searchable” so that they utilize the indexes appropriately for maximum performance and minimal disk reads.

A few weeks ago I got called out of my coding lair and was brought into a meeting with management where they informed me and 5 other people that there was going to be a friendly internal competition. A web application for tracking portfolios, initiatives, projects, and deliverables had been thoroughly designed for desktop\tablet and phone use. There would be two teams, each consisting of 1 QA and 2 developers. We were to stop our current work and devote our time and efforts on this, and in 1 week the two teams would reveal what they had done. Management would take care  of clearing our schedules. At the end of the week, the implementation that won would go on to being used for real by our group (about 120 people) and possibly other groups within the organization.

We were allowed to use any technology we wanted. The requirements (if we could meet them) were that the web application used our organization’s login accounts, was behind SSL, worked on a desktop and tablet (which had one design) and phone (which had a modified design). We were given high-fidelity mockups and access to the designer and the main user, and we were told to come back in a week and show how much we got done.

The requirements

I can’t post any screenshots because I don’t have permission, but I will describe the application a little. The main page of the application showed a timeline of all the projects for all the initiatives for a given portfolio. There may be 10 to 15 initiatives and each initiative may have 10 projects. Projects are 1 to 12 months and have usually a few deliverables. There are about 12 portfolios. The timeline on the main page has different colors for each deliverable indicating status, which can be modified via the main page if the user has permissions to do so.

There are three levels of access in the application: View, Edit, and Admin, and users are scoped to the particular portfolios that they should have any access to. Admin users can add other users and assign portfolios and roles. Users are added just by entering the user’s existing internal organization account username and the web app will import the user information from LDAP.

Editors can add and modify initiatives and projects, which each have their own screens. Initiatives have names, descriptions, year, and a few other fields. Projects have name, description, start and end dates, overall budget, dependencies on other portfolios, labor calculator (list of individuals and their bill rate, percentage engagement, number of days engagement, etc with dynamically calculated labor rate total), deliverables (with name, are delivery date) and a few other fields. Adding and removing labor rate line items and deliverables is in-browser, not a screen refresh, and the deliverable dates are scoped to be between the start and end dates of the project.

There is also a summary screen that is similar to the timeline page except it show all the initiatives and projects for a portfolio in sort of a spreadsheet view where you can collapse and expand initiatives to show or hide their projects. The budget totals are also shown for initiatives and projects.

That’s the gist of this application. It is non-trivial and had a very rich design and interaction. My team had an excellent QA, excellent front end dev, and me who was the only one who knew MarkLogic. The other team chose to implement theirs using a Javascript front-end architecture communicating with CouchDB (later Java with MongoDB) on the backend. The two teams involved very skilled people. If these two technology approaches were going to go head-to-head, these were the people to do it.

The results

So how did it turn out? We deployed our implementation to a server behind SSL hooked up to our auth scheme in my organization. We implemented all the requirements, and then some. We had three experiences fully implemented: desktop\tablet, phone, and dumbphone (meaning no CSS or javascript and limited functionality) which the server would pick for you based on detected device abilities. We had SOAP\XML and REST\JSON services. We had the entire application translated into 9 languages: English, Spanish, Russian, Greek, Japanese, Korean, Chinese, Arabic, and something else (I can’t remember).

We also ran the application through a series of speed, load, penetration, and injection tests. Average HTML page size was 1 or 2 KB. Average response time for HTML was 250 ms (assets were all cached after the first hit). With 1000 concurrent users, there was no perceived change in performance. There were no failures or warnings with session hijacking, cross-site scripting, malicious code injection, malicious character injection, or any other security test. This was using an old-ish former Apache server which was not even half-power for a MarkLogic recommended setup. One ED Node, no caching.

We did this all in one week.

The other team did not finish. They had a few pages that showed some data, but it was not deployed, didn’t have authentication, you couldn’t create or change data, no phone or other experiences, at least based on what they showed. No other languages than English either. No security, penetration, load, or injection tests. For whatever reason, that’s how things ended up.

What does it prove?

To me, this demonstrated hands down the speed at which you can implement non-trivial, fully featured, mature, enterprise-class, performant, and secure web applications very quickly, in only one week, with only three guys, sometimes.

So that application my team created is being used for real now, and it has had some bug fixes and other tweaks, especially with IE (did I mention it was cross-browser compatible?). Oh and did I mention the build and deployment time is less than 30 seconds?

It’s hard to argue with end results. People’s time is expensive, and the less time you need to spend on people creating frameworks, building stacks, adding in security and other enterprise stuff, the better. And with all the claims and arguments that get tossed around, all I really think is: scoreboard.

Everyone wants to get their query times low, and there’s a lot of satisfaction knocking time off and running tight and lean. After some tinkering, I got my query time for this mobile device detection down to 0 seconds. That’s Zero. It was a moment of Zen for me. Seeing the profiler say 0 seconds is just not the same as 0.001 seconds or anything else. In theory of course it took some time, but having the calculation come out to 0 for a very non-trivial feature got me thinking: how many other features can I get down to 0 seconds?

What if all the features of a web page took 0 seconds? Impossible? I have written several applications and the target server time for generating a page is around 250 milliseconds, or 1/4 of a second. This includes checking for authorization, querying the data, formatting the data, dynamically generating the HTML, and even translating it into a foreign language, all on the fly. So I know 250 milliseconds is possible because I’ve done it repeatedly. I also realize that going from 500 milliseconds to 250 milliseconds is easier than going from 250 milliseconds to 0 milliseconds.

But processors and RAM are getting faster, SSD drives may totally change the time profile of data retrieval, MarkLogic will itself get faster, as will web application code. So at some point, someone is going to write a web application where the server time to generate the HTML for it will take 1 millisecond. That will be faster than the TCP overhead of the computers trying to negotiate a connection to each other.

Gone are the days of pages taking 10 seconds to load or your visitor goes away. Now the expectation is that server-side HTML response is sub-second. For my stuff, I aim for 250 milliseconds, and that’s very doable, and I often seen times of 60 to 70 milliseconds (under load with 1000 concurrent users, on one box, checking auth for each request, etc. etc). But now I’m thinking that’s too long. I’m going to expect sub-100 millisecond pages for now, but I’m going to ultimately be looking to cross the 1 millisecond boundary, but a lot of that will come from hardware improvements.

Maybe this is like the sound barrier or the 4 minute mile, where it sounds crazy until it is actually done by someone, and then it’s the new standard. For dynamic, non-trivial web applications, I don’t think any technology would be more likely than MarkLogic to be able to pull it off. And I don’t think it’s really that far off in the future.

Happy coding.