Extra-Cubicular

"But the consumers at home are also our employees at work. And when they arrive at their desks, they bring a new set of expectations that have been shaped by their experiences with the Internet, cell phones, email, mobile hand-held devices and iPods. These and other innovations have changed the way they consume and interact with information."

- Daniela Barbosa in "The Taxonomy Folksonomy Cookbook", available at http://solutions.dowjones.com/cookbook


One thing is certain: if Information Technology departments continue to perpetuate the status quo - if they continue to conduct business in the way they always have - the problems I discussed in my previous post will be exacerbated; the general utility of IT will continue to decline. In order to break the cycle, IT must change from within. As I said before, this is psychology, not technology.

1. IT must adopt a decentralized approach and architecture.


Out of all of the recommendations that I am making, this is probably the toughest change to make. Changing technology is easy. Changing thought patterns is hard. Nevertheless, it is the essential starting point.

This step is predicated by the realization that the act of processing information not the sole purview of IT. Everyone does, in some way or another; IT just happens to use formalized systems to do so. The structure of the IT department needs to reflect this - IT must become more of a "gravitational centre" than a fortified embankment. I suggest a layered approach with decreasing privileges:




• Core IT staff focus on providing api's, security, data models and business rules.


• Core developers working on corporate-wide applications; ones that will be used by nearly everyone in the organization.


• Developers working on specialized apps that require read-write access to the core databases.


• Developers working on specialized apps that add or improve on existing data.


• Developers working read-only apps, local annotations, and reporting.


This model pushes development activity out toward the end users and allows IT staff to concentrate on caring for an 'information ecosystem' that will better serve their organization.


2. Push information processing tasks toward the user.


• Utilize department-level programmers. These are the people who are outside of the purview of the official IT department, yet still know how to build basic applications. In any organization of appreciable size, they're already there and they are eager to show what they can do. Allowing them to work will alleviate much of the load on the core developers.


• Give users generative tools that will allow them to solve their own problems. Just like the availability of word processors destroyed the notion that typing was something that happened inside of the typing pool, appropriate tools would destroy the notion that programming is something that happens exclusively inside the IT department.


• Graft onto existing, known tools. Organizations are run via ad-hoc, user-created spreadsheets and documents. We need to figure out how to use these documents as interfaces into systems, rather than containers of data. For example, if a copy-and-paste operation could be treated as a pointer to a bit of data, a document created through a series of these operations would become a living document that would update itself whenever those data sources changed. Users have caught on to what copy and paste does; they will understand the difference between that and a pointer to live information, given the right training.


• Training. We do live in the Information Age and ignorance of the basic tools of such cannot be accepted. In addition to basic computer knowledge and system-specific training, some users need to be trained on high-level information processing issues. Things like:
  
  • How to recognize when a task should be automated.
  
  
  • What kinds of tasks can be automated.
  
  
  • How to use the tools available to them to solve their own problems.



3. Flexible tools & systems.


In the most excellent article (Ontology is Overrated), Clay Shirky points out how using rigid categorization systems to organize information often doesn't work very well. I believe this conclusion applies to most top-down methods of organization. Real life just doesn't fit into neat compartments and real-world data doesn't cleanly fit into predetermined categories. Trying to predict at design-time all possible uses of an application's data is a fool's errand. Systems must be designed to not only accomplish their primary objectives, but to also:






• Allow for annotations, tagging & user-generated customizations.


• Make all data searchable from outside the application.


• Allow data to be remixed.


• Allow data to be easily extracted.

A few years ago Nicholas Carr wrote "Does IT Matter?". While I feel that the provocative title is somewhat misleading, I have to agree with his conclusions. As it is practiced today, IT is too inwardly-focused, too unwieldy and unresponsive. Most IT departments seem to think that the next multi-million-dollar rollout is going to be the silver-bullet panacea that will fix their problems. That just isn't going to happen. IT's current problems are systemic and to paraphrase Albert Einstein, they won't be solved with the same level of thinking that created them.

"History never repeats itself; at best it sometimes rhymes."
- Mark Twain
"Those who cannot remember the past are condemned to repeat it."
- every history teacher that I've ever had

We now stand twenty-five years into the PC revolution, give or take a few. It has fundamentally changed how we work and play, do business and government. It was the platform that enabled the communications upheaval that we call the Internet. Yet, despite everything that has changed, the IT department at your average company has remained fundamentally static. Sure, the machines in the glass cages have quad-coloured logos instead of blue-lettered ones. Sure, the company's desks are populated with devices that can actually process information, rather than mindlessly display green characters. Sure, different technologies are being used, but the mission, mandate and structure of your IT department has remained the same: to be the keepers and providers of all computer-related technology in the company; a centralized point that all information-processing activities flow through.

When computers first entered the business world, they were behemoths that required constant care, feeding and grandmothering in order to limp along. Thus, the IT department was born. The people responsible for the care of the original beasts were, by necessity, very sharp, extremely intelligent and very anal about their children. They had to be. Before lithograph techniques could abstract away logic gates and transistors, every bit meant another wire attached to multiple tubes (later transistors) and every single one of them had to work. Even one malfunction would bring the whole apparatus to a standstill. Thus the requirement for intelligence and thus the drive to keep control of the entire system.

Today, IT departments can only be described as beleaguered. User demands are skyrocketing – exponentially. Organizations are collecting more and more data and the users want that data re-formatted, related, reported, re-purposed, in more locations, on more devices and so forever on. In the face of these demands, the typical response is to circle the wagons. A standard answer of “six months and $50,000” is given to even the most basic requests, in the hopes that those requests will simply go away. In practice, the requests usually do go away, although the requirement that sparked them doesn't.

This has left users to get on as best as they can. In the face of unresponsive IT departments, users create ad-hoc documents and spreadsheets and plug in vital corporate data. They have to copy and paste information between multiple applications. They create and set up “rogue” bits of infrastructure – programs, databases, servers. They co-opt database fields intended comments for other data. Entire careers are spent moving chunks of data from information silo to another; the digital equivalent of manual labour. It doesn't take long for an organization's users and its IT department to be enclosed in two different, yet intersecting spheres of data.

To be fair, the blame for this situation cannot be placed solely at the feet of IT. Computer knowledge is the only domain I know of where people are actually proud of their own illiteracy. Supporting these users - dealing with those who seemingly refuse to learn the tools they use on a daily basis - puts IT staff on the defensive.

The history of computing has cycled between open, generative systems and closed, locked-down controlled environments. Until very recently, the enterprise world has defined the state of the art, but that is no longer the case. The freewheeling cowboy-coders that brought us Web 2.0 have shown us a better way of handling information – one that is closer aligned to how that information is used. To paraphrase Bruce Schneier, information-handling is a critical task that is intertwined throughout all of our lives, and it is one that is too important to be pushed away and left in the hands of a small circle of experts. IT must make a fundamental change that goes beyond switching tools or services. This is psychology, not technology. For an IT department to be worthy of the title, it must go beyond being stewards of systems and take a holistic view of how information is used throughout the entire organization.

Regarding the killer app for the semantic web – I think that it's a mistake to look for one. The semantic web is a 'foundational' technology – a building block. It is highly generative, but doesn't provide much value until something is built with it. The layperson will look at and ask “What use is it?” - a sentiment that I heard expressed in reference to both personal computers and the Internet in their early days.

I hope that a killer semantic app doesn't emerge. If one does, that app will dictate the style and flavour of the first generation of semantic web applications. One huge success will spawn hundreds of imitators, but they will only be imitators. Only after those imitators fail (and/or coalesce) will other ways of using semantic web technology be considered (a pattern that I've seen repeated many times).

Rather than a single 'killer app', the semantic web first needs to permeate existing applications and systems. This will form a primordial ooze of rdf data that will allow a 'semantic ecosystem' to emerge. The value promised by the semantic web is predicated by a mass of available semantic data. If that mass of data doesn't appear, or if access to it is tightly controlled by restrictive policies, the semantic web will be hamstrung and won't likely have much of an impact. Fortunately the age of openness is upon us and the exponential value of open api's and data is readily apparent, so I don't think that this will be a problem.

Labels: semantic web

Email sucks. It's an information black hole. Information goes in, but doesn't come out.

OK - here's an example of what I'm talking about:

From: MyBoss
To: Peon Programmer
Subject: project update

I looked over the site and I am impressed by the progress you have made. Please make the following changes:
1. Increase the font size of the text boxes.
2. Change the background color of the second page to blue.

Pty H. Boss

Now suppose that I don't get around to making these changes when I get this email. A few days later I go searching for project revisions. Here's a small portion of what I see:

+ Today
[ 11/11/2007 ] Check out this video!!!
[ 11/11/2007 ] Weekly meeting update
[ 11/11/2007 ] Developer Den Newsletter
[ 11/11/2007 ] Today's Specials at the Hardware Hut
[ 11/11/2007 ] U free for lunch?
+ Yesterday
[ 11/10/2007 ] HTML Hacker's Missive for 11/10/2007
[ 11/10/2007 ] Company News
[ 11/10/2007 ] Group meeting today at 3:00
[ 11/10/2007 ] Shipment # 947104 is in

... and that's assuming that my spam filter is 100% accurate. This is a rather contrived example - in reality each day's list is easily five to ten times as long and we're all tracking multiple projects and situations instead of just one.

Email has become the default method of information transmission because it is expedient and ubiquitous. However email seems to be the point where information looses it's context - where a bit of information looses it's connections to other bits of information that gives it relevance. All of these chunks of disconnected data are then presented to the user in a homogeneous mishmash of spaggetitized factoids.

Everyone's inbox is chock-full of data, but none of it is accessible as information. Stand back an arms' length from your inbox. What can it tell you about a specific project or situation? Pick a message at random. How does that message relate to any of the others? These questions cannot be answered but through manual means. This information to do so is there, but it is obscured.

We need a new messaging paradigm - one that takes into account relevance and context. RSS is a good start, but when you have the 37 Signals guys still picking email over RSS, it shows that RSS still has a ways to go. Hopefully things will change soon.

Labels: Observations

Well, after reading Saul Griffith's column “Who wants to be an Inventor” in the latest Make magazine, I got inspired. Here's an idea that has been echoing around in my head for quite some time. Enjoy, critique, criticize – it's all good.

Digital authentication schemes are currently based on four things: what the user knows, what the user has, what the user does, or what the user is. Yet there is something else we can use: the user's history (I couldn't think up a catchy “what the user ...” saying for this one). We use our interpersonal histories to recognize each other in our day-to-day relationships. Imagine talking to someone who looked and acted exactly like a good friend, but had no knowledge of any of your previously-shared experiences. Authentication based on history has become a Hollywood staple – usually when some malevolent lifeform has taken over someone else's body. How does the hero find out if someone is friend or foe? By asking about their history.

Enough theatrics. Here's how it works:

1. When the user's account is set up, the server generates a block of strings. Each string is a series of random characters sufficiently large enough to be difficult to guess, and the block is long enough to protect a suitable number of the user's sessions. The server stores this block, and the user receives a copy as well.

2. The first time the user attempts to authenticate, the server randomly selects one position out of that user's block and requests that position from the applicant. The applicant's client looks in the user's block and sends the string residing in that position to the server. If the string supplied by the applicant matches the string held by the server, the applicant is let in. The server then generates another string and sends it to the user. The user's client replaces the previously requested string with the newly-generated one, and acknowledges. The server then performs the same replacement, and notes the position that was used.

3. The next time the user attempts to authenticate, the server requests two strings from the applicant: the one at the position used in that user's previous session and another one chosen at random that has not been used before. Upon successful authentication, the strings in both positions, in both the user's block and the server's block, are changed to random ones generated by the server.

4. From then on, when a user attempts to authenticate, the server requests two strings from the applicant: one that was used in the user's previous session (this will actually be the string generated during the user's last session) and another string that has not been used yet. Each position is changed upon successful authentication.

5. If the strings supplied by the applicant do not match the strings stored by the server, access is denied. The account should then be locked and steps should be taken to authenticate the user through manual means (telephone call, ID check ...)

6. When each position has been used, the server can either:

   1. Forget which positions have been used and start the process over again, using the same block of strings. (least secure, but easiest)

   2. Re-start the process by using the same block of strings in the same order. This would require the server to remember the order in which the strings are used.

   3. Re-initialize by creating another block of strings and sending the block to the client.

The chief advantage of this scheme is that it gets stronger the more often it is used and the more places it is used in. Regular password-based authentication gets weaker with more use and more locations because every login represents an opportunity for the user's password to slip into someone else's hands. For example, suppose a user's home computer had become compromised with a keylogging trojan. Once that user logs into a password-protected web site, that account is compromised. Both the legitimate user and the intruder are then able to access that account. Then we can only hope that the intruder moves on before doing too much damage, the user changes the password, or the server detects simultaneous logins and acts on it.

However, if this scheme was in place the damage would be prevented, or at least mitigated. Suppose that some nefarious individual managed to obtain a user's login credentials, including a complete copy of the user's string block. One of two scenarios would occur: if the user logs in before the intruder does, the legitimate string block is updated, rendering the intruder's copy obsolete and useless (remember that the server will ask for the string given out in the user's previous session). If the intruder logs in before the user does, he will be able to access that account until the legitimate user comes back. At that time the account will be frozen, and the user's login credentials will have to be re-established (after manual authentication checks are done, of course)

This scheme actually performs two tasks – authentication based on information previously exchanged, and it detects compromised accounts. It would be best used as an extra layer of security, run in the background after a regular password check, to create a two-factor authentication scheme.

The primary disadvantage of this scheme is that it gives the user another security credential to manage. The user's block of strings would have to be stored somewhere (encrypted, of course) and would have to move around with the user. A USB key would be the natural choice, but that comes with a host of it's own problems (lost, stolen, file corruption). However, when compared to other two-factor authentication schemes out there, this one is nearly as secure as a SecureID token, but without the hardware expenses, and better than fingerprint or facial recognition, which are unreliable.

Labels: Ideas

Entrepreneurs and system designers are both admonished to ensure that their creations address a specific need. "See a need, fill a need" as Bigweld would say. Yet this vision of development needs to be compared to the two greatest revolutions in information-handling of our time: the personal computer and the Internet. Initially, neither of them addressed a specific need. Let me re-state that: when they came into prominence both were solutions in search of a problem, and were described by many as such. However both of them gave users the tools they needed to solve their own problems.

The rest, as they say, is history.

Mitch Kapor (wikipedia bio) is famous for saying "Architecture is Politics". I never really understood what he meant until now.

In today's organizations, information is the currency of power. System architecture determines who can access what data, how, where and when. That is politics. It is embedded into the design process itself. Every system begins it's life as a document that specifies what that system will do and how it will do it. Great effort is expended at this point in an attempt to 'cover all the bases'; forecast all possible ways the system will be used. The underlying assumption is that the ones doing the designing are the only ones who will determine how that system will be used. That concentrates the decision-making and access to information in the hands of a few, and that is politics.

This approach comes with many problems, not the least of which is that the capabilities of the system are limited to the imagination of it's designers. No team, regardless of the abilities and brilliance of it's members, can predict all the ways a particular system, or the data it contains, could be used. Current development methodologies demand that they try, and when they fail, the system becomes a limiting factor to it's users rather than an enabler.

A core principle of Web 2.0 is "Harnessing Collective Intelligence". This needs to become a core principle of corporate applications as well. Everyone in a typical organization, from delivery boy to president, has information that could be potentially useful. Capturing and processing all of this information would be a major strategic advantage for that organization.

Labels: Observations