February 2009 Archives

I was talking with @fyip on Twitter yesterday, and he told me that he installed the Geneva Framework and Server on a Windows 2003 Server machine. I asked what he had to do to get the installer to work on a non-Vista, non-Longhorn OS. He said that he installed Orca, changed the LaunchCondition VernsionNT value from 502 to 501 as show in the following screenshot:

I did the same thing on XP, and it installed great:

Onetime Preparations

Next came the fun part: spelunking through batch scripts to figure out how to configure things, so the samples could run and the install could be verified. To this end, I installed the Windows Server 2003 Administrative Tool Pack which includes the only version of certutil that runs on XP. I also had to download choice.exe from the Windows 98 Resource Kit (yikes!) which one of the batch scripts depends on. Then, I opened a Visual Studio 2008 command prompt, CDed into "C:\Program Files\Microsoft Geneva Framework\Samples\Utilities\Scripts." I then popped open SetupCert.bat that's in that directory, and changed line 80 from this:

pushd "%ALLUSERSPROFILE%\Microsoft\Crypto\RSA\MachineKeys"

To this:

pushd "%ALLUSERSPROFILE%\Application Data\Microsoft\Crypto\RSA\MachineKeys"

Then, I ran that script passing localhost as the name of the cert's subject and root as the store name like this:

SetupCert.bat localhost root

I was prompted to overwrite the existing cert that had a subject of localhost (which got there IINM by running the script a couple of times as I tried to get it working).

Next, I had to setup the SSL configuration in IIS. I did this by opening the IIS management applet, selecting properties from the context menu of the default web site, and clicking the Server Certificate button on the Directory Security tab. In the wizard, I opted to assign an existing certificate and chose the one issued to and by localhost.

Finally, I ran iisreset.

Running the Samples

I tested out the "Simple Claims Aware Web Application" sample included with the SDK. To start with, I needed to create the virtual directories. I took a chance and ran the CreateSampleVdir.bat in that sample's directory fully expecting it fail. To my surprise, it seemed to work. Then, I popped open the 2008 edition of the solution and compiled it. I alt-tabbed over to my browser and entered in https://localhost/PassiveRedirectBasedClaimsAwareWebApp/. This resulted in an error saying that The current identity (MyMachineName\ASPNET) does not have write access to 'c:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files'. I fixed this by running this command:

    C:\WINDOWS>C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\aspnet_regiis.exe -ga MyMachineName\aspnet

I then was presented with this error:

Parser Error Message: ID1024: The configuration property value is not valid.
PropertyName: serviceCertificate
Error: ID1039: The certificate's private key could not be accessed. Ensure the access control list (ACL) on the certificate's private key grants access to the application pool user.

So the changes that I made to the SetupCert.bat script above didn't seem to work. To fix this, I opened "C:\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\MachineKeys". Which of these was the private key? Luckily, I still had the console window open where I previously ran SetupCert.bat. At the end of that script, it said which one was the private key. I opened the properties dialog box for this file, and granted the ASPNET user read and execute permission. Once I applied these settings and hit the Relying Party (RP) in my browser again, I was prompted for my Windows credentials. I entered them, and was logged into the STS who shipped me back to the RP where my correct identity was displayed!

Conclusion

This was a heck of a lot of work just to run on XP. I sincerely hope that Microsoft ends up supporting XP when the Geneva Framework RTMs. If you have an IT department that won't support anything but XP or have customers that require this older OS, but want to develop with Geneva, please let Microsoft know. Go to the connect Web site, and provide them with feedback saying so and/or post a comment on the Geneva forum.

[Edited 5/10/9: Be sure to also read Vittorio's detailed explanation about what proof keys are.]

In my last post on Geneva-related terminology, I intentionally postponed defining the term proof key. It is an important one, so I wanted to dig into it a bit more before trying to define it. Since then, I've reread Michele Bustamante's article wherein she discusses the different types of tokens and I've also clarified my understanding with Hervey Wilson, architect for the Geneva product line. After doing so, this is my current understanding of proof keys. 

A proof key is a cryptographic key that allows you to create a digital signature. There are two types of tokens used in WS-Trust and WS-Federation:

  • Bearer tokens
  • Holder-of-key tokens

Bearer tokens do not have proof keys while holder-of-key tokens do.

The following process describes how the proof key included in a holder-of-key token is used to ensure that messages sent from subject to RP are indeed from the subject claiming to have sent them. This protocol-level protection guards against man in the middle attacks:

When creating RSTRs, the STS encrypts both the SAML token and the proof key within it using the public key of the RP. The STS also includes the proof key in the RSTR and encrypts it using the public key of the subject. Upon receiving this, the subject is able to decrypt the proof key. It then sends the SAML token and a message that it signed with the proof key to the RP. When the RP receives the message, it is able to decrypt the SAML token using its private key. It computes the signature of the token using the public key of the STS and compares it to the one included in the token. If they match, the RP knows that the token was sent by the STS and that the proof key in the SAML token is trustworthy. It then uses this to sign the message. If it matches the signature created by subject using the proof key, the RP knows that the message originated from the subject claiming to have sent it and no one else.

The anatomy of the RSTR was depicted in Michele Bustamante's recent article in MSDN about creating a custom STS which is reprinted below for conveyance:

This is all wonderful, but there's a problem: every passive client (i.e., Web browser) requests bearer tokens from the STS. Thus, the RP has no way of using the protocol described above to protect against man in the middle attacks. Why? Because Web browser doesn't know how to retrieve proof keys out of RSTRs. Even when an Info Card selector is used, a bearer token is still created. (It seems to me that the selector could get the proof key out of the RSTR and sign the message with the proof key before sending it to the RP; however, from what I understand, this isn't done for reasons unbeknownst to me.)

Because passive, browser-based clients can only work with bearer tokens, they need a way to protect against man in the middle attacks. This is done by ensuring that all communication that takes place during the authentication process happens over SSL. Stop and think about that for a moment. What type of credential is being provided by the user? A username and password in many cases. When a managed Info Card is being used, even that is often secured using a username and password. So, tell me, what is the difference between all this and basic auth over SSL? Nothing. But this question (which my team and I asked ourselves) presupposes that WS-Federation Passive Profile was intended to provide a protocol with a superior level of security than its alternatives. This is not its intention. WS-Federation Passive Profile is intended to enable advanced federation of services (authentication typically). It does this using the same level of security that is trusted and accepted by many other software applications relying on basic auth over SSL. In conclusion, the absence of a proof key in a bearer token isn't a shortcoming of the protocol; it's a limitation of current browser software that leaves the RP susceptible to man in the middle attacks. This threat vector can be mitigated using SSL.

Many companies are just coming to grips with Web 2.0. This "upgrade" has been costly for many organizations in terms of training, realignment to changing market demands, associated hardware and software upgrade requirements, as well as other related impacts. Despite the recent growing pains, however, organizations must begin preparing for the fast approaching third version of the Web. Currently, there are many predictions about what this next major release will entail, but most agree that it will center on cloud computing, mobile devices, increased bandwidth capabilities, and something called the semantic Web. Though the features included in Web 3.0 are important, the actions that businesses take in preparation for its inevitability are just as critical if not more so. As Marshall (2007) of Search Engine Journal claimed, the eventual ability of computer programs to sift through mountains of data and intelligently present pertinent results to their users will require businesses "to have a better understanding of who their target audience is, what they want, and how they think about what they want." The impending incantation of the Web will dramatically affect businesses' spending, strategies, markets, resources, and day to day operations; it will even render some businesses obsolete. If organizations fail to plan or postpone preparations, when the third version of the Web is fully realized, they may be too late to satisfy unmet customer needs, capitalize on emerging trends, and seize new opportunities. 


Many are predicting that we will eventually be surrounded by computer terminals of all sizes and shapes. These portals will grant us access to the Web where we will use it to do many of the same things that we use it for today. We will do some of these things differently, however, and will utilize this new version of the Web to perform currently unthinkable activities. These conduits to the Web might be things like flat screen displays built into tables, car stereos that learn your tastes by connecting to on-line databases and stream the results to your auto, and cameras built into amusement park rides that send photos of riders to email addresses encoded in their admission bracelets. Enabling these types of things will require these "dumb terminals" to connect to some larger machine with sufficient computing power and storage. This model is reminiscent of the mainframe era except that, as Kelly (2007) prophetically pointed out, the reinvented mainframe is a single computer that spans the entire earth. This gargantuan super computer is the result, Kelly said, of connecting so many PCs, handhelds, cell phones, notebooks, and servers. Using these interconnected machines as a single super computer will provide computational capabilities necessary to enable currently unimaginable applications. This radical, sci-fi-sounding notion is referred to by mainstream media and techno-bloggers as cloud computing because the interconnection of computers is done via the Internet which is often described as a cloud.

 

Though Web 3.0 is only here today in nascent form, this fundamental part of it is already available from an increasing number of service providers including Amazon, Google, GoGrid, IBM, Microsoft, and others. These businesses have created data centers of machines and sell access to them. Most use a per-use payment model where consumers are only billed for the minutes that computers are allocated to them. These armies of worker machines are available to anyone via the Internet. The availability of an almost limitless amount of storage and computing power available on a pay-per-use basis is a phenomenal game changer. As Nelson (2009) suggested, "[u]sing the network as your business platform is really interesting in terms of helping to level the playing field between large and small businesses" (as cited in Chan, p. 18). Startups no longer require capital investments to purchase the computers necessary to compete with larger, established market leaders. By taking advantage of this new paradigm, organizations can be "more effective and efficient" in many aspects compared to those who use established approaches to IT Nelson goes on to say.

 

Many large established corporations are leery of cloud computing though. With a lot to lose, they are reluctant to accept the associated risks. The advantages are significant; however, the fall from grace is far. As large enterprises seek to remain relevant though, they must take this paradigm shift seriously; they need to begin investing in R&D, training, and pilot projects to overcome the inherent risks associated with cloud computing. They need "to examine [themselves] closely, measuring the risk against when and where cloud computing can be effective," Schwartz suggested (2008, p. 11). Everything, he predicted, "will eventually be available as a cloud service" though not every resource a business has to offer "will be accessed from the cloud" (p. 11). Now is the time for large enterprises to begin determining which of their assets should be accessible from the cloud and which should not. Nelson suggested that this new computing phenomenon will be utilized by "the large enterprises, even with the regulatory concerns in the equation" (as cited in Chan, p. 18). If their acceptance comes too late or not at all, however, smaller companies enabled by the leveling quality of cloud computing could potentially uproot them from their current positions of dominance.

 

Though these cloud terminals will be pervasive, there will be times when we do not have immediate access to them. Unable to tolerate even the briefest moments of interrupted access, consumers will drive the invention of innovative mobile technologies that will keep them constantly connected. These handheld computers will form a major component of Web 3.0. Devices of unimaginable sizes, shapes, and designs will connect us to the cloud, allowing us to do amazing acts that will further revolutionize our culture and way of life. The transformative impact of Web 1.0 and 2.0, as profound as they were, will be dwarfed, I believe, by the impending tsunami of portable devices that is about to wash over humanity.

 

By connecting to the cloud, devices will have enhanced capabilities far beyond what they have been able to do thus far. This tie will allow them to combine the Web's data with real life. The information and computational power of the Web will be coupled with the telephones, cameras, camcorders, microphones, GPS receivers, compasses, keyboards, and styluses embed in mobile devices. "Forward-thinking brands have to come to appreciate that more [people] will consume digital content via mobiles" insisted Woods (2008, p. 37). It is imperative, he urged, that companies "start thinking beyond the PC-based internet experience" (p. 36-37). This marriage of mobility and the Net will result, as Kelly said, in an embodiment of the Web. Many "artifacts" in the natural world, he predicted, will have some form of "webness" in them. This combined with a further proliferation of cloud-connected handheld devices will create a 4th dimension of sorts wherein the physical world is combined with the Internet to create a new reality. Two examples of this today are Enkin for Google Android mobile phones and Microsoft Tag.

 

As Web 3.0 approaches, we will see a dramatic increase in broadband speeds. By many reports, the average American downstream broadband speeds today are less than one megabyte/sec. For example, a company called In-Stat polled 700 high speed Internet customers at the beginning of last year and reported that the average download speed was 3.8 megabits/sec or 0.475 megabytes/sec. Reid (2008) said that the Web's next version will include "connection speeds comfortably in excess of 10" megabytes/sec (p. 14). This one order of magnitude increase will have a revelatory effect on what and how originations are able to conduct their businesses. This speedup will allow companies to transmit large datasets like video and voice as never before. This phenomenon may allow consumers to perform searches verbally via mobile devices, participate in video conferences using portable tablets, or watch HDTV on cell phones.

 

The last aspect of Web 3.0 (covered in this paper at least) is its ability to be mined by computer programs for information that can be presented to users in a more intelligent manner. This capability will be made possible by the reprogramming of Web sites to include standardized data that can be read and interpreted by machines. Using this additional code, machines will be able to understand not only what the page's content is (as in Web 1.0 and 2.0) but what that content means. This additional level of interpretation is why this aspect of Web 3.0 is referred to as the semantic Web. All search engines to date have used what van Harmelen (2007) refers to as "location finding rather than information retrieval." Current, mainstream search engines find places on the Web and organize them based on "keyword matching, keyword density, and ranking," says Marshall. Given a query, the search engine presents the user with a list of locations that it believes best satisfies the search. It leaves the last and most critical step to the researcher: determining which of these locations actually have the pertinent information being sought.

 

The semantic Web will change all this. It will dramatically reduce the amount of time we spend performing "location finding" and also change how we search for information. The amount of time we spend researching as we do today will be reduced, and autonomous search agents will perform queries for us, presenting results in various contexts. These programs, Woods predicted, "will make the internet seem more intelligent [by] extracting knowledge that other people put into it in a way that looks intelligent" (2008, p. 35). As Marshall foretold, the result will allow researchers, consumers, and businesses to "more easily find products and services that provide precisely what they need." This will force brands, as Marshall also said, to know exactly who their customers are and what they want.

 

As we move into this era of increased technological sophistication, businesses will be confronted with immense challenges and opportunities. Thriving will require advanced hi-tech abilities; however, it will also demand that every organization clearly define its mission and vision by continuously answering the following questions posed by Drucker (1967): "What is our business;" who and where is our customer; what is he buying; and what does he consider value (p. 77, 80, 82, 83)? By continuously answering these questions and planning early, businesses will ensure that they remain relevance and successful in the era of Web 3.0.

 


 

References

 

  • Chan, C. S. (2009). Cloud computing - levelling the land. Enterprise Innovation, 4 (6), 18, Retrieved from Business Source Complete database.
  • Drucker, P. (1967). The effective executive. New York: HaperCollins Publishers.
  • In-Stat (2008, March 5). US consumers eyeing "high speed" broadband more closely. Retrieved February 18, 2009, from http://www.instat.com/press.asp?Sku=IN0804264MBS&ID=2255
  • Kelly, K. (Speaker). (2007, December). Predicting the next 5,000 days of the web. TED Talks. [Webcast]. Retrieved February 18, 2009, from http://www.ted.com/index.php/talks/kevin_kelly_on_the_next_5_000_days_of_the_web.html
  • Marshall, M. (2007). The semantic web and its implications on search marketing. Search Engine Journal. Retrieved February 18, 2009, from http://www.searchenginejournal.com/the-semantic-web-its-implications-on-search-marketing/5390/
  • Reid, A. (2008). The evolution of the internet. Campaign (UK). 19, 14, Retrieved from Business Source Complete database.
  • Schwartz, E. (2008). The dangers of cloud computing. NetworkWorld Asia, 4 (7), 10-12, Retrieved from Business Source Complete database.
  • Van Harmelen (2007). Knowledge management with the WWW. Retrieved February 18, 2009, from http://kogs-www.informatik.uni-hamburg.de/~neumann/WMA-WS-2007/WMA-11.pdf
  • Woods, A. (2008). Is web 3.0 a load of !#*!? No, read on.... Revolution, Retrieved from Business Source Complete database.
As someone just getting started with Geneva Framework, I have had to learn out a lot of new terms.  The curve associated with this is steepened by the fact that many of the terms used in the framework, WS-Trust, and WS-Federation are synonyms of each other.  In some cases, the two standards refer to the same things differently (different authors, publication dates, knowledge bases, etc.).  Because the Geneva Framework has support for both protocols, it uses terms from each and even creates new, generic ones to describe shared notions.  For example, WS-Federation uses the word realm (and wtrealm) when speaking about the relying party to which a security token is being requested of the STS.  Conversely, the WS-Trust spec uses the term AppliesTo for this idea.  Geneva Framework uses these is some places and also uses the more generic term scope to convey this concept.  This overloading of terms takes some time and digging to wrap your head around.  To help with that, I present my best understanding of some of the different terms you'll encounter when working with the Geneva Framework. 

  • ActAs - An optional element that can be included in an RST to indicate to the STS that claims are being requested by a subject who is acting as someone else.  The result is the ability to delegate authority to another party.
  • AppliesTo - The URI of the relying party which is used to check that the requested security token is to be created for a trusted RP.
  • Audience URI - The name of the party that is the meant to receive an issued token.  It is used to determine if a token is intended for the receiver or not.  This is sometimes called the audience restriction condition as well.
  • Claim - An assertion made by an issuer about a subject as a triple {A, B, C} where A is the dialect, B is the claim type (a unique string that's typically a URI), and C is the right or claim value.
  • OnBehalfOf - An optional element of an RST that indicates that the requestor is making the request on behalf of another.
  • Realm - A term that comes from WS-Federation that means the same things as AppliesTo (from WS-Trust)
  • Scope - A generic term that means the same thing as AppliesTo (from WS-Trust) and realm and wtrealm (from WS-Federation).

This isn't all of the terms you'll run into, but it's a start.  I'll add more, like proof key, as I figure them out.

In a previous post, I promised to blog about a bit more of the details of MarkLogic, a native XML database.  Not wanting to break my word, I've finally put my thoughts on "paper."


The incipience of the company and its database stems from a need in the marketplace for a unified platform that provides an integrated search engine with the features typically associated with database management systems.  It marries the advanced capabilities found in the former (including indexing of semi-structured data, fast querying of information stored in desperate formats, searches based on Boolean logic, stemming, and thesauri) with typical features of database systems (including role-based security, clustering,  and others).  This synergy was pioneered in 2001 by people from Google and other successful companies that are famous in these two domains.

 

This pairing is part of what provides the following features of MarkLogic Server:

  • The ability to secure content by roles
  • Content stored in the database is accessible via a RESTful Web service interface or a .NET programming library similar to ADO.NET which they call it XCC
  • The server provides a WebDAV interface as well allowing Office applications, for example, to open documents/content directly which can facilitate ad-hoc reporting requirements among other things
  • Ability to import Microsoft Office documents, PDFs, Web pages, and other documents types into the database which can then be searched and exported to alternative formats
  • Support for the W3C's XQuery language
  • XML indexing that provide fast full-text and semi-structured searches
  • An architecture that can purportedly scale to Internet-levels (hundreds of terabytes)
  • Support for searches within XML content using stemming, thesauri, and spell-checking
  • Transactional updates

In traditional RDBMSs, text and B-tree indexes are disconnected.  As a result, they are constantly out of sync.  To keep them aligned is costly.  If these resources aren't able to keep up with demand or synchronization isn't done, searches won't find content that is stored in the DB even though it's there.  Keeping the two types of indexes in sync at the cost of CPU and disc resources eventually hits an upper limit of what the computer can provide.  As a result, large systems are limited by the RDBMS from scaling beyond this threshold.  This design is only a limitation in the theatrical sense for small loads but can become a real issue when the size of the database increases to enterprise- or Internet-levels.

 

Many RDBMSs currently support an XML data type.  For example, Microsoft SQL Server, Oracle, and DB2 have such a feature; however, not all RDBMSs store XML natively.  This distinction is made based on the way that the information is serialized.  Native XML databases do not store the information in a relational backend.  Oracle originally insisted that all XML could be shredded into a relational form and that native XML storage wasn't needed.  IBM disagreed and saw that they needed to store XML in a non-relational form, and did so even with the first version of their product that offered XML support.  Oracle has since changed its opinion and now stores XML in a non-relational form in its current release. 

 

This reversal of opinions makes MarkLogic look very good because they saw that this shredding process and relational backend storage design was antiquated and would not work for document content as it did for last few decades with tabular data.  Unlike traditional database engines, MarkLogic does not use a relational storage system and, thus they do not shred XML into this form.  Instead, verbose XML content is converted to a logical representation which is compressed and serialized.  This compression and decompression is costly in terms of time and CPU usage, so MarkLogic has striven to find a good balance between the amount of compression needed and the quantity of storage space consumed. 


MarkLogic, as of version 4, has support for XQuery 1.0 (with or without it proprietary extensions) and is backward compatible with older versions of the draft standard.

 

MarkLogic has a number of customers in various industries including publishing, government, and others.  Their first customer, Elsevier, is the largest publisher in the world.  With their very first installation, they were able to meet the publication giant's requirement to store terabytes of data while simultaneously being able to query it between 10 and 100 times per second. The Elsevier deal cost many hundreds of million of dollars.  [Update: I talked to David Rees of MarkLogic today (Mar. 10, 2009) and he assured me that the deal with Elsevier was competitively priced and far, far less than what I originally reported.  I apologize for the mistake and for the misinformation.]

 

Since their initial offering, MarkLogic has since deployed a solution for many other customers including a Dallas-based company that needed to snoop at messages that went in and out of mainframe computers from many different sources.  This company had to store this freeform data in a repository that would allow for fast retrieval and analysis.  This customer tested DB2 and MarkLogic, but the latter eventually won out because of its ability to ingest massive amounts of XML data with optimal speeds.  DB2 was able to store the information without issue; however, as the database grew, the time that it took to ingest additional amounts of information was prohibitive.  Conversely, the time that it took MarkLogic to insert new data grew at a much slower rate and was almost independent of the amount of content under its control.

 

If MarkLogic is able to outperform traditional databases like this, why aren't other vendors changing to speed up their products?  The reason is because their architectures and approaches are difficult to alter considering their age and install base.  With a chance to start from scratch in a world where discs and memory were cheap (relative to the 70s where the roots of many RDBMSs were architecturally fixed by their ancestor the System R), MarkLogic was able to create a system that didn't discount ideas that required larger amounts of disc usage or memory.

 

Some of the ways that MarkLogic diverges from traditional systems in this regard is that it does in-line updates.  This allows them to cache modifications in memory and to do sequential I/O on disk.  This in-memory cache is indexed in an efficient manner using techniques that aren't possible in traditional engines that can't use such caching mechanisms.  When a delete is performed in a MarkLogic server, a flag is set in memory indicating that the record has been removed; however the information on disc isn't removed at the same rime.  Rather, a background process periodically finds the records that have been marked for deletion and (if configured to do so), removes them from disc.  If a piece of content has been updated in the cache but is subsequently deleted before this background process has run, the update is never reflected on disc and the records are removed.  Thus, I/O is reduced and requests are fulfilled more quickly.

 

The memory that MarkLogic uses for its cache is journaled and can be recovered if the system crashes before this background processes has had a change to flush it to disc.  It is also sensitive not to store too much data in RAM to limit resource pressures on the system and to ensure that restoration times after crashes aren't prohibitively time consuming.  The journaling mechanism employed by MarkLogic is purportedly less costly than that of other RDBMSs.

 

This approach also allows for data to be queried even if it has been deleted.  If the database is configured not to remove the information from disc but to simply mark it as deleted, queries can be made over information that was previously removed.  The net effect is the ability to "time travel" through old data (if you will).  This notion is similar to a recycling bin that is never emptied and isn't capped in size.  The result of course is that more disc storage must be available, so the usefulness of this feature must be weighed against the storage costs.

MarkLogic is a really compelling technology that is certainly worth investigating, and I hope this little writeup has played a small part in that process.