Table or Booth

Google's Invisible Hand

"So far they have only demonstrated excellent intentions, but the invisible hand of the market is quite a thing, and you often find it stuck right up your ass, or in your pocket looking for your wallet"

- Paul Ford, about the Semantic Web, Google and their increasing control over the dissemination of internet data.

It makes me wonder: will Google's dominance inevitably become a dangerous thing?  Or will the invisible hand eventually demand safer alternatives as well?

Paul Ford is a riot.  Check out ftrain.com if you've never been.

It Must Be Official

Consider tagging officially on the business radar.  CNN is even reporting it.

Opting for Opt-in

"News and updates" pollute our mailboxes every day.  They're not always the infamous kind of spam we all know and hate; now they're 'newsvertisements,' sent to you because of that little, afterthought-checkbox at the bottom of on-line account sign-up forms.  They're usually checked by default.  At the rate new web apps are created, this is an increasingly common occurrence.

Keeping it Fresh

"Selling web-based software through ISPs is like selling sushi through vending machines"

- Paul Graham, in Hackers and Painters, Big Ideas From the Computer Age.

He makes a good case for owning the hardware as well as the software.  From a startup perspective, I can see how it would be tempting to do otherwise and consider outsourcing this.

Semantic Web Introduction

Occasionally, I get asked for pointers on the Semantic Web.  Since I've made embarrassingly brief appearances as a representative to a W3C Semantic Web Working Group and have followed the Semantic Web evolution for several years now, I  suppose I can rightly point and say "Go there."

In doing this, a technical question that keeps popping up is... "What is it?"  The, slightly-less-technical follow-up question is "Where can I go for more info?"    Consider this blog entry answers to these innocent questions. 

Eclipse Con 2005: Google on Scaling

Googles scaling strategy was integral to their search dominance and later emergence as a major player in this net-based software era.  This is a live blog of the Google keynote, presented by Urs Hoelzle at Eclipse Con 2005.  It briefly explains how they approached and solved the scaling problem.

They use cheap, commodity hardware, not high-end servers.  Lowers cost/CPU allowing for greater redundancy.  But, this increases potential maintenance.

They run Linux.

To win, they planned to fail; seems to break basic rules for success but actually it's smart.  When you have hundreds or thousands of servers, expected hardware failure at any rate makes efficient responses to these failures less than trivial. 

Urs took us through a few unbelievable, humorous slides showing their hardware progression from the late 90s on.

Redundancy is a Google core value.  Not losing data is central to Google's business so, it makes sense.  Requires reliable infrastructure building blocks.  To achieve this Google realized several useful abstractions:

Google File System (GFS)

• GFS Master manages metadata; these are replicated
• 64 MB file 'chunks' are managed Chunkservers, also replicated 3X
• Chunks also triplicated for fault tolerance. 
• GFS client servers directly access the GFS Master and Chunkservers

Basic Computing Cluster

• Needed massive parallelization and distribution that are easy to use
• MapReduce solves the problem.  MapReducing = mapping + reduction.

Map: take input k/v and produce set of intermediate k/v pairs
Reduce: emit final, condensed k/v pair - these are sorted, merged search results

MapReduction is so redundant that, in one unplanned test, they lost 90% of their reduction 'worker' servers and all of the reduction tasks still completed.  Now that's fault tolerance!

Regarding Query Frequency, he showed several successive graphs where frequency of
"eclipse" searches, before Eclipse.org, peaked every three years
"world series" searches peaked every year
"watermelon" peaked during the summer

Funny but, more importantly, Google uses these patterns to learn from the data.  This learning process is broken into two basic steps: establishing relationships between searched data, then clustering the related documents for relevant search results.

It was an interesting talk.  Their scaling approach isn't mind bending but it's sooo effective.  What's most fascinating to me is that they had the audacity and forsight to tackle the problem at the beginning.  For more information on How Google Works, go here.

Eclipse Con 2005: Seamless UI Integration

This was a good topic by Julian Jones, the usability 'gatekeeper' for IBM's Eclipse tools.  The PDF has details but here's a quick summary:

• Don't harden UIs early.  Consider them loose and changeable until close to the final release.  Note: This implies clean model-view-controller separation and keeping a light view.
• Try out divergent ideas by shipping them in milestones (or the equivalent).  This encourages "natural selection" of good UI patterns.
• Use 'centralized planning' only where it is truly necessary.  Example: Eclipse's unified Help TOC.

On enforcing tight UI integration between disparate plugin development groups, Julian suggested that the dictatorial approach will never work and to use a 'gatekeeper' for publicizing good UI examples.  I agree.  IBM publicizes new UI features using formal walkthroughs.  He sights the PDE Overview page as an example of a good, internal UI-pattern that grew organically this way. 

The gatekeeper also manages decisions regarding ugly UI overlaps.  Currently, there is an ongoing discussion about how to resolve GEF sliding-pallettes, icon-bars, and side-bars in 3.1M5, which all effectively do the same thing but target vastly different users.

Eclipse Con 2005: Tim O'Reilly Keynote

Tim's perception of technology patterns is truly remarkable.  Here's a distilled list of operational principles he offers for gaining competitive advantage:

• Embrace the value chain: Proprietary applications -> build on -> Commoditized technology -> build on -> Single-source suppliers.
• Design for Participation by architecting for easy incorporation into larger system.
• Design for Usability by releasing early and often, where users submit bugs and solution suggestions.
• If your business is being commoditized, focus on testing, assembly, and integration so user's can have the best of the market's commodities.
• Give users choices but not too many by offering products in proven configurations (standards).  As new configurations emerge support them.
• Since today's applications are internet based, treat them like services (not artifacts) and add features as a part of the normal user experience.  Tim calls this "The Perpetual Beta".
• Add value to your product by incorporating user-data.  Good example: amazon  Bad example: Barnes & Noble.
• Set aggressive defaults for aggregating and incorporating user data to improve the overall value of the application.  These can be unobtrusively configurable.
• Design applications for n-devices.  Today's software is above the level of a single device.
• Reduce complexity and barrier to entry for third party developers by explicity supporting emerging classes of applications (see PHP, Ruby on Rails).
• Be the single-source supplier for an essential subsystem of a bigger, open system.  Examples: Intel -> the CPU, Cisco -> Internet, Navteq -> Mapping
• If you can't own the data, own the registry.  Examples: Google -> Searching,

Tim started the presentation by quoting Jeff Hawkins, the author of On Intelligence saying "All the brain knows are patterns."  It's patterns like these here that will determine the very success or failure of today's technology companies.  If you can think of others or have comments, please add them here or ask Tim.

Agile vs. RUP - Practicing Resonance: Part 2

While Agile and Unified Software Development-style processes  claim to be iterative and adaptive rather than defined and predictive, they both have limitations when it comes to efficiently producing software in medium sized teams.  This particularly true when these span multiple geographies.  However, they can both teach us how to achieve what I'm calling 'resonance' (see part one of this series). 

Note that the Rational Unified Process (RUP) is the official IBM refinement of what is essentially the Unified Software Development Process (Jacobson, Booch, Rumbaught).  As background, I'll first proceed with this embarrasingly incomplete review of the Agile and RUP practices.

The RUP is "use-case driven, architecture-centric, and incremental and iterative".   Each iteration is part of four overall 'phases': Inception, Elaboration, Construction, and Transition.  Iterations occur in each phase.  Activities in iterations are focused on one of the four activities: gathering requirements, analyzing, designing, implementing, and testing.  Each of these activities place a more or less important role as the project moves from phase to phase.  As Craig Larman points out, RUP is not the dreaded 'waterfall' approach.  It's the mis-application of its myriad of optional workflows and documents that tend to promote that illusion.  Here are some personal observations about how it works in practice.  RUP practitioners are:

• document-centric - they labor to produce detailed documents whose prescribed purpose is the necessary outcome of many of the activities in the overall process
• perfectionists - since they move only once from phase to phase, it is imperative that the artifacts (code, documents, etc) they produce be perfect.  In theory, iterating within the phase should allow for mistakes and trend towards more and more perfect artifacts.
• cautious - RUP practitioners cannot afford to design or implement a mistake because it will destroy the integrity of the overall plan that was established during or even before the first phase.
• independant - some might unjustly say "isolationist".  The truth is, they trust the RUP process 'framework' within which they expect the right information to be available in order to  get their specific job done.  Well defined interfaces between all the components, allow each person to work in relative isolation and then later plugin their individual pieces.

Agile development is more of ethic than a process (the founders refer to it as a  "manifesto").  To get a feel for what a completely "Agile" process might look like, take a look at Xtreme Programming (XP).   In essence, to be agile in software development is to quickly and appropriately react to new information in the development cycle in a way that best satisfies the customer.  To best make this happen, Agile practitioners are:

• customer focused - they insist on having the customer highly involved in each aspect of what is being produced.  The customer's regular feedback is immediately incorporated into the development effort, even it if means significantly changing what had already been produced thus far.
• real - they don't try to make guesses and purposeful keep the scope of their efforts small.  They explore often to get the information necessary to continually make small steps forward.
• intuitive - they rely on experience to guide them through the process and try to minimize activities that 'just don't make sense' for the task at hand.  If a comprehensive design specification doesn't get the job done, then they don't attempt it
• communicators - they try to 'peer program' as much as possible.  They try to convene strategic and tactical thinking in the same activity by having someone 'drive' while their peer 'navigates'.

The "independant" and "communicator" characteristics in  each of the previous lists are the ones most worth focusing on in order to achieve resonance.  Part three will consider RUP and Agile practices' techniques for dealing with independance and the basic communication issue, the very heart of resonant practice.  Most importantly, we will consider exactly what this means for geographically disperse teams.  Stay tuned!

Agile vs. RUP - Practicing Resonance: Part 1

Galloping Girdy was the first bridge over The Tacoma Narrows in Washington State, built in the 1940.  It was the third largest suspension bridge in the world.  It was only the "...first bridge over..." because it tragically collapsed from massive vibrations created by a freak wind, only four months after it was built.  It's quite an interesting bit of history and never ceases to remind me of the importance of good engineering. 

Powerful vibrations or their 'resonance' provides a good metaphor for addressing issues resulting from the developing trend in software development to outsource.  Pacific Northwest National Laboratory defines resonance as "the state of a system in which an abnormally large vibration is produced in response to an external stimulus, occurring when the frequency of the stimulus is the same, or nearly the same, as the natural vibration frequency of the system."  For my purposes, resonance can be restated as "a state of existence in which the maximum amount of useful knowledge is transferred from one person to another because they are 'in sync', producing the highest potential for efficiency in an organization."  Even more simply, resonance is just good communication.

With software jobs going overseas and the punishing market conditions of the post-bubble, commercial software professionals are challenged to deal with an increasingly dispersed development geography while also continuing to maintain or improve the pace of innovation.  None of the predominant software development processes ('heavy', 'agile', whatever) is perfectly equipped to handle the load.  In part two, I'll talk about the two most popular processes, focusing on their benefits and drawbacks as they relate to resonant practices.  In part three, I'll conclude with a short description of how these processes might be tweaked to get closer to resonance.