GUEST POST from Steve Todd

My  experience with innovation mentoring during 2013 turned into something quite different from what I was aiming for. The goal was initially innovative new products, but the result was leadership development.

During this month I ended two different global mentoring engagements (one co-worker in Russia and one co-worker in China). In each case we had a wrap-up meeting to discuss our progress against our original goals.

The Russian use case is the best example of setting goals in one area and achieving them in another.

My mentee (Alexey) and I set a goal of innovation in his area of expertise (compression algorithms). In one sense our intent could roughly be described as an effort to stimulate the creation of new high-tech products.

In order to achieve this goal, we discussed a set of new behaviors that could stimulate technology interchange with people outside of Alexey’s general circle. By proactively increasing his network, marketing his technology, and following up on opportunities, we had hoped that he could end up with a new product proposal.

This new product proposal may indeed some day happen.

In the meantime, at our closing meeting we discovered that the mentoring helped Alexey cross the bridge from an expert in his field to a leader in his community.

For example:

• Alexey held an innovation training seminar to share innovation approaches (e.g. innovation by adjacency).
• Alexey also held an educational seminar on his area of expertise.
• These seminars were held during the announcement of EMC’s global idea contest in April 2013.
• These trainings, held early in 2013, can be correlated to three Russian award winners (out of 28 global winners) at this year’s EMC Innovation Conference.
• In each case, the three winners displayed a pattern of the innovation by adjacency approach.
• Alexey formed relationships with research leaders outside of his region (e.g. Brazil).
• Alexey reached out to collaborate with the local sales team to understand customer requirements better.
• Alexey attended a one-day workshop in Israel to better understand the Telco environment.
• He became more involved with local leadership in his facility and more active in the monthly technology councils.
• He decided to be more disciplined in the area of intellectual property generation.

In other words, Alexey became more publicly visible outside of his comfort zone, and began to tackle tasks that were outside of his traditional scope.

This increase in scope and visibility is a path that leaders trod.

It will be interesting to trace his innovative output going forward. The link between innovation and leadership is well documented.

The mentoring session with my Chinese co-worker (Diego) took a similar route. At the end of a 10 month mentoring engagement Diego had increased his visibility and scope.

As these two mentoring engagements ended, some new ones are beginning (Ireland) that I hope will generate similar forms of insight.

Steve

https://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Fellow

My  experience with innovation mentoring during 2013 turned into something quite different from what I was aiming for. The goal was initially innovative new products, but the result was leadership development.

During this month I ended two different global mentoring engagements (one co-worker in Russia and one co-worker in China). In each case we had a wrap-up meeting to discuss our progress against our original goals.

The Russian use case is the best example of setting goals in one area and achieving them in another.

My mentee (Alexey) and I set a goal of innovation in his area of expertise (compression algorithms). In one sense our intent could roughly be described as an effort to stimulate the creation of new high-tech products.

In order to achieve this goal, we discussed a set of new behaviors that could stimulate technology interchange with people outside of Alexey’s general circle. By proactively increasing his network, marketing his technology, and following up on opportunities, we had hoped that he could end up with a new product proposal.

This new product proposal may indeed some day happen.

In the meantime, at our closing meeting we discovered that the mentoring helped Alexey cross the bridge from an expert in his field to a leader in his community.

For example:

• Alexey held an innovation training seminar to share innovation approaches (e.g. innovation by adjacency).
• Alexey also held an educational seminar on his area of expertise.
• These seminars were held during the announcement of EMC’s global idea contest in April 2013.
• These trainings, held early in 2013, can be correlated to three Russian award winners (out of 28 global winners) at this year’s EMC Innovation Conference.
• In each case, the three winners displayed a pattern of the innovation by adjacency approach.
• Alexey formed relationships with research leaders outside of his region (e.g. Brazil).
• Alexey reached out to collaborate with the local sales team to understand customer requirements better.
• Alexey attended a one-day workshop in Israel to better understand the Telco environment.
• He became more involved with local leadership in his facility and more active in the monthly technology councils.
• He decided to be more disciplined in the area of intellectual property generation.

In other words, Alexey became more publicly visible outside of his comfort zone, and began to tackle tasks that were outside of his traditional scope.

This increase in scope and visibility is a path that leaders trod.

It will be interesting to trace his innovative output going forward. The link between innovation and leadership is well documented.

The mentoring session with my Chinese co-worker (Diego) took a similar route. At the end of a 10 month mentoring engagement Diego had increased his visibility and scope.

As these two mentoring engagements ended, some new ones are beginning (Ireland) that I hope will generate similar forms of insight.

image credit: biography.com

Sign up here to get Human-Centered Change & Innovation Weekly delivered to your inbox every week.

GUEST POST from Steve Todd

In my last post I described how a new form of application workload began gaining steam last decade: infrastructure-based applications.

The complexity of these applications caused a tremendous surge of innovation in data center environments. This surge continues to this day and is often referred to as a “Software-Defined Data Center” (SDDC).

The diagram below classifies applications (and their workloads) into three different categories.

This diagram is meant to convey the following:

• There are three categories of “data” that applications send to the IT infrastructure: content, metadata about that content, and infrastructure metadata related to the operation of the business.
• There are three types of applications generating different categories of data: traditional applications that are focusing on content and metadata (e.g. an ERP application), applications that are focused only on infrastructure metadata (e.g. an interface to configure a storage system), and hybrid applications that are concerned with both the infrastructure and the content (e.g. a backup application).
• The IT infrastructure must be flexible enough to handle all three types of content in a coherent way.

What does “flexible” mean?  If we zoom in one one of the applications, we see that the applications have a wide variety of choice in their method of data storage and retrieval:

Traditional applications can write content and metadata to the IT infrastructure using an ever-evolving array of protocols: block, file, object, database, key-value pairs, etc.

The other two applications (hybrid and infrastructure-based) have similar choices.

This was (and is) the state of the industry. Designing an IT infrastructure to handle this phenomena was tricky indeed. New applications began to surface in the area of backup/recovery, security, content management, and infrastructure management.

In future posts I will dive down into each of these areas to highlight how these solutions evolved.

image credit: emc.com

Sign up here to get Human-Centered Change & Innovation Weekly delivered to your inbox every week.

GUEST POST from Steve Todd

In previous posts I have described how application workloads drove innovation in the areas of block (SAN), file (NAS), and object(CAS) storage systems.  I discussed the evolution of these systems in the context of the increased importance of metadata, and used the diagram below to highlight the different roles these three systems take when it comes to metadata management.

In my last post I claimed that new forms of metadata would challenge the underlying infrastructure even further (and result in new forms of innovation).

For this post, I’d like to describe one of the main areas where evolution in metadata expansion occurred: infrastructure-based workloads.

A wide variety of infrastructure-related applications (IRAs) began to surface as a result of the proliferation of block, file, and object storage systems within a data center.  These IRAs, like their “traditional” application counterparts, had a desire to read and write their own “IRA data” to a robust and performance storage system.  This “IRA data” can be thought of as metadata falling into one or more of four categories:

1. Configuration Management.
2. Data Center Security.
3. Backup and Recovery.
4. Content Management.

In order to more fully understand the impact of these applications within the data center, it helps to extend our diagram above to include these new applications alongside the “big 3” storage systems.

Infrastructure-based metadata, in general, has much more complex inter-dependencies than the more traditional content-based metadata.  While content-based metadata often has a cardinality of one, infrastructure-based metadata can map to multiple pieces of content and multiple pieces of other infrastructure-based metadata. Consider the diagram below:

Traditional application metadata might contain metadata which enriches other pieces of content.  In this example, patient metadata further describes content stored in an X-RAY, electronic medical record, and doctor’s notes.  The cardinality of the relationship of application metadata is a “zero or more relationship”.

Infrastructure-based metadata, on the other hand, has much more complex cardinality. Consider backup metadata. This form of metadata not only has to maintain cardinality with metadata and content, but it also has to consider other infrastructure-based metadata, such as information about the data center infrastructure (where is my backup device), security-based metadata (who is allowed to access), and/or compliance metadata (is there an audit process or workflow going on)?

Each type of infrastructure-based metadata has this type of complex cardinality of zero or more.

It’s this exact issue that caused a tremendous surge of innovation, and I will highlight some of these innovations in future posts.

image credit: emc.com

Sign up here to get Human-Centered Change & Innovation Weekly delivered to your inbox every week.

GUEST POST from Steve Todd

I started writing about application workloads and innovation by using a simple diagram:

Sign up here to get Human-Centered Change & Innovation Weekly delivered to your inbox every week.

GUEST POST from Steve Todd

This series of blog posts has focused on the evolution of high-tech infrastructure in response to constantly-evolving application workloads.

In my last post I described how unstructured and metadata-rich application workloads drove the rise of Network-Attached Storage (NAS).  The diagram below allowed me to highlight differences between block and file system architecture.

Unstructured content benefits from metadata association. NAS systems provided the binding between the two.  The approach used by many vendors involved the interspersal of content and metadata within a disk array infrastructure.  Block-based systems of that era, on the other hand, viewed all blocks as “content”, and had no fundamental awareness of application metadata. The overlay below highlights this difference.

The NAS approach of tight interspersal of content and metadata became a hurdle for a new class of application workloads. To quote my EMC colleague Stephen Manley, these new applications wanted to do “even cooler” things with their metadata.

For example, applications wanted to:

• attach increasingly larger amounts of metadata to content.
• create formal ontologies for metadata (e.g. XML rules for metadata structure).
• search through metadata at high speed.
• enforce policies on content via metadata keywords (e.g. retention periods).

The increased importance that these new workloads placed on metadata drove the industry to treat metadata as a first-class citizen. The “interspersal” technique used by most NAS devices did not lend itself to the new workloads.

As a result, the industry evolved (yet again) in response to these new applications and facilitated the rise of object-based storage systems.

Object-based systems allow applications to “attach” rich metadata to content and bind them together via an object-identifier. Under the covers, object-based storage systems were not constrained to intersperse the metadata and the content. They could be stored as separate entities, which “freed” the metadata to be used in more diverse and beneficial ways.  In fact, the content itself was “freed” from the linkage to a specific directory, which facilitated new levels of sharing and collaboration for content.

The implementation of object-based storage systems also gave vendors the opportunity to address additional shortcomings that NAS-based systems were experiencing at the time, including file size maximums and file count limits.

The first object-based implementation was termed content-addressable storage, or CAS. Wikipedia provides the definition of CAS below:

a mechanism for storing information that can be retrieved based on its content, not its storage location.

The diagram below highlights CAS function and operation in the context of one of the first CAS implementations (known as Centera):
Instead of using the traditional file-based access methods (e.g. file open, read, write, and close), the Centera approach allowed an application to write a random stream of data, associate it with relevant metadata, and store it as a package to the Centera storage system. In return the Centera system would return a unique identifier to the application.

This approach caused a fundamental shift in application architectures, which enabled:

• A permanent binding between file content and an unlimited amount of metadata associated with the file content.
• The removal of responsibility for “where” the application placed data. The application no longer had to specify a logical directory location for each file.
• Object counts could scale into the billions, well beyond the limit of many file system capacities at the time.
• The metadata contained keywords to implement policies (such as how long to retain a document and disallow deletion).

A third access pillar was added to the data center as a result of new application workloads. Many customers deployed all three access methods: block, file, and object.  Capacity-based, object workloads are graphically depicted in the lower-half of our workload framework. Some object-based workloads required high service levels (e.g. hospital applications) while some did not (e.g. YouTube).

As a result of all three types of application access methods (block, file, and object), data and meta-data continued to grow unabated within customer data centers. This gave rise to a new problem: the growth of new forms of metadata related to the data center operation itself.

I’ll cover “The Rise of Metadata Part 2” in my next post.

image credits: universalmachine.com; stevetodd.com

Sign up here to get Human-Centered Change & Innovation Weekly delivered to your inbox every week.

GUEST POST from Steve Todd

The evolution of application workflows is a major driver for innovation in IT infrastructures.

I have written several blog posts about the hypothesis above. In order to prove this point, especially in regard to storage system IT infrastructure, I have written several articles describing:

• The limitations on the amount of application workload a single disk drive could support in the 1980s.
• Storage system innovation that resulted from these performance workload problems.
• The service level requirements of application workloads from the 1990s that also resulted in significant innovation.
• The emergence of multi-departmental application workloads in the late 1990s that caused the emergence of shared storage and SAN approaches.

I spent a good part of my career writing software that handled application workloads for VNX (block-based I/O).  In particular, I wrote a lot of software that performed caching algorithms and RAID algorithms inside of a cached disk array.

As part of my experience handling application workloads (e.g. block read and write requests) I was handling raw bytes of data and didn’t often consider the topic of metadata.

In a recent conversation with EMC colleague Stephen Manley, we discussed the rise of metadata in the context of the application workloads of the 1990s. More and more applications began to emerge that focused on the management of raw blocks of data (otherwise known as files). By associating metadata with the raw content, applications realized the following benefits:

• The ability to store data, and access it, in a more logical fashion.
• The ability to easily share it by creating metadata with access rights.
• The ability to protect content from unauthorized use.
• The ability to create multiple copies (e.g. active, backup, and compliance copies) and to know where those copies are.
• The ability to create workflows around the content and share it with the right people at the right time.

One of the most significant innovations in response to the metadata trend was NAS: Network-Attached Storage. NAS is predominantly about metadata management, or, as Stephen Manley likes to say, “knowing information about the data you are storing so you can do really cool things with it”. The value is in the metadata. It dictates the accessibility to the content.

In the same way that application workloads drove the industry from physical disk drives to cached disk arrays, the rise of metadata drove the industry from local file systems to network-attached, shared storage. An example side-by-side view of block and file storage highlights this point.

The deployment of block and file I/O systems into customer data centers eventually drove the industry to unified architectures supporting both block and file. One notable innovation that resulted was a hybrid  approach known as MPFS. This innovation allowed an application workload to write using a file protocol, but transparently read using the BLOCK protocol. This approach provided, for example, a 3-4x performance increase over traditional file system techniques (the industry eventually adopted this innovation into an industry standard approach called pNFS).

Due to the surge in the generation of unstructured content, the NAS market exploded. In some cases, application workloads began to exceed the capacities (and capabilities) of file system technology, pushing the industry toward a new paradigm: OBJECT.

Applications desired to associate increasing amounts of metadata with their content, which stressed the existing approach for interspersing metadata and content. These workloads began to push the industry deeper into the realm of capacity-oriented workloads, which required further innovations in the IT infrastructure. The diagram below highlights this push down the Y-axis.

Workloads pushed capacity-oriented infrastructure in two directions (as highlighted by the diagram above). Some applications began storing massive amounts of metadata and content with high service levels (e.g. fast and available X-ray retrieval during a hospital procedure), while other applications had less rigid availability and/or performance requirements (think YouTube videos).

In either case, application workloads desired to do more and better things with their metadata. This phenomena gave rise to a new class of storage system: object-based storage.

image credit: murraystate.edu

Sign up here to get Human-Centered Change & Innovation Weekly delivered to your inbox every week.

GUEST POST from Steve Todd

Evolving application workloads have been a primary cause of innovation in the high-tech industry. For example, applications drove the need for faster disk response times in the 1980s, leading to innovations such as the cached disk array (Symmetrix) and RAID5 (CLARiiON).  In the mid-1990s applications drove the need for higher service levels, resulting in technologies such as snap copy (Symmetrix TimeFinder) and remote replication (Symmetrix SRDF).

As a review, consider the diagram below that stimulated this discussion. In place of Symmetrix and CLARiiON I have used the descendant names of these products in 2013 (VMAX and VNX). These two products continue to position themselves in the upper right hand quadrant of the workload chart, providing both high performance and high service levels.

Application workloads continued to play a role in the late 1990s as well. The adoption of technologies such as cached disk arrays, RAID, snap copy, replication, etc., resulted in a shift towards a shared storage model. Instead of one application workload pointing at one storage device, enterprise customers began configuring multiple application workloads, from different departments within the enterprise, against a shared storage device.

For example, the engineering department might use a portion of the storage for test and dev, the sales department might use a database for customer leads, and the marketing department might create a file share to store marketing collateral such as PowerPoints and PDF files.

In this era, the requirement to handle multiple application workloads gave rise to an innovative new paradigm known as the Storage Area Network (SAN). In order to build a SAN, innovation was required by several sub-components of the overal SAN solution. These sub-components are described below.

Storage Switches

Different departmental workloads ran on different departmental servers. In the 1990s the hypervisor had not yet been invented. Some storage devices (e.g. CLARiiON) did not have enough ports to connect to multiple departmental servers. As a result, the industry moved towards a switch model in which the servers connected to the switches, and the switches in turn connected to the storage ports. This type of consolidation of the storage network is best typified by offerings such as Connectrix.

Connectrix helped propel the trend to storage consolidation. By the late 90’s open systems customers were challenged by the proliferation of Windows servers,  each with their own storage. Some customers complained of needing a grocery store shopping cart every Monday morning to collect the disks that failed over the weekend! Symmetrix, VNX, and Connectrix capitalized on the  availability  and administration  problems of a decentralized storage architecture. A level of service well  established in the mainframe storage space won support with open systems.

Server-based Multi-pathing

Existing applications still desired high service levels, and the introduction of a switch in between the server and the storage introduced new failure permutations that had to be overcome. In addition, multiple I/O paths now existed which could be leveraged to provide even higher performance levels via load-balancing techniques.

In 1998 a company named Conley was working on an innovative approach to solve these problems. They were acquired by EMC and the product was eventually branded as PowerPath. PowerPath became a staple of the industry; millions of copies were sold. It was an example of application workloads driving storage innovation up into the server level.

As a result of the introduction of switches and PowerPath routing software, application workloads continued to transition further and further away from the storage devices, as evidenced by the diagram below.

Multi-tenancy, Fairness, and Trust

The introduction of SAN technology created a scenario whereby multiple tenants (departments) were “occupying” a portion of the underlying storage device.  This introduced the problem of fairness (predictable allocation of storage resources to each tenant) and trust (fencing and/or prohibiting a tenant from accessing the resources of another tenant).

The problem of trust was solved via innovations such as zoning (at the switch level) and lun-masking (at the storage level). Over time masking and zoning were implemented in a variety of different locations in the SAN stack.  The issue of trust within the IT infrastructure would grow significantly more complex moving forward (I will cover this topic in future posts).

Management and Orchestration (M&O)

Perhaps the key area that began to take center stage from an innovation standpoint was the management of the SAN at each layer (the server, PowerPath, switch, and storage perspectives). This became increasingly important as customers began to install hundreds of applications upon hundreds of servers and point them at dozens of switches and storage devices.

One of the significant innovations at the time was the capability of “pushing” server information (server name, operating system, application information, and port information) down into the storage system. This marked one of the first times that applications began “registering” themselves with storage devices, another strong indicator that application workloads were driving storage innovation.

For example, in the diagram below, Servers A-D would register themselves with the underlying storage system. This enabled a management interface (for example the Unisphere tool) to associate a departmental workload (running on a given server) with a portion of the disks in the storage system.

File as a Workload

The list above is large and doesn’t do justice to the long list of innovations required to make the SAN vision a reality (for example I haven’t mentioned the advent of Fiber Channel technology).

More importantly, I haven’t mentioned another critical requirement that departmental workloads placed on the storage infrastructure: a desire for serving file-based and block-based requests together from one infrastructure. At that point in time, block storage systems and file storage systems often existed side-by-side as separate systems.

I’ll spend some time discussing this workload in a future post.

image credit: murraystate.edu

Sign up here to get Human-Centered Change & Innovation Weekly delivered to your inbox every week.

GUEST POST from Steve Todd

I have been blogging my way through the theme that application workloads drive innovation into the underlying high-tech infrastructure. In my previous two posts I gave evidence that the disk array industry was itself launched by the evolution of application workloads. Both posts focused on the performance aspect of application workloads (the upper portion of the Y-axis displayed below).

For this post I’d like to highlight that innovation in high-tech is also driven by the service level needs of applications (the X-axis).

In my first post I mentioned that the Y-axis is a bit confusing. The units at the top (I/O performance) don’t match the units on the bottom (storage capacity). In a similar way the X-axis is a bit vague. Some would argue that the X-axis represents increasingly strict levels of governance, risk, and compliance (GRC).  Others would say that it represents the requirements on availability of IT infrastructure services (e.g. five nines, or downtime of less than six minutes per year).

At a high level, as the X-axis extends to the right, the application places increased criticality on protecting and accessing the correct application information. As it extends to the left, attributes such as correctness and availability become less critical to the application.

These service level requirements, historically, have driven innovation in the same way as performance requirements. In order to demonstrate this point, I’d like to pick up where I left off in the last post.

The First Disk Array Innovators

Symmetrix innovated with the introduction of a cached disk array. The resulting product was extremely popular, causing a “gravitational pull” of data onto more and more Symmetrix systems.

In a similar way, CLARiiON innovated with the longest-lasting implementation of RAID5, and likewise customers began storing increased amounts of critical data onto the product.

Over time, users of both products realized that the lifeblood of their business was information, and they began to place very high service-level demands on their information infrastructure. Their disk array cabinets were holding massive amounts of disk drives, thus increasing the likeliness that one (or more) of those disks would fail.

Both product teams responded to evolving application workload service level requirements in two very different ways.

Timefinder and SRDF

In the 1990s the increased amount of disk drives on the customer’s premise increased the odds of a double disk failure, which could result in lost customer data. Standard practice at the time was tape backup and restore.

Some customers, however, wanted a higher service level in terms of the amount of time it took to get back up and running with a restored copy of the data. This workload requirement caused the Symmetrix engineers to innovate by creating local copies of the data inside the Symmetrix itself. These copies, known as BCVs (business continuance volumes) were part of a product offering known as TimeFinder, and could be used as a much faster backup and restore mechanism. Over the years TimeFinder has blazed a continuous trail of innovation and solved a number of other application service level use cases as well. TimeFinder copies have also been used for disaster recovery testing and validation, point-in-time recovery, database consistency checks, and test/dev of offline data.

This innovation is often referred to as Snap Copy (or Snapshot), and is depicted below.

In addition to making local service level copies of data inside the array, the SRDF innovation (Symmetrix Remote Data Facility) allowed customers to make remote copies as well. This innovation provided a high service level to application workloads that often had to enable the entire business to be up and running at a remote site in the face of an outage at the primary site.

Clearly, application workload service level requirements drove innovation in the case of the Symmetrix technology. The SRDF innovation is depicted below.

In the mid-range, service levels drove the CLARiiON product to accelerate the implementation of the industry’s first mirrored write cache.

Restore as a Workload

Often times during the workload discussion we think of applications such as financial trading, medical imaging, and database applications. Each of these represents a workload with distinct characteristics in terms of I/O rates, bandwidth/throughput, and service levels.

What is often left unstated is that the backup/restore process is a workload unto itself.  And in the case of CLARiiON, the “restore” workload ended up accelerating the deployment of the industry’s first mirrored write cache.

The first deployment of CLARiiON came with a warning: “don’t use RAID5 for write-intensive workloads”. The sweet spot for RAID5 applications would be 70/30 read/write ratio (or anything with a read ratio higher than that). Unfortunately, for customers restoring data to a RAID5 configuration, the restore application itself represented a single-threaded 100% write workload! Restore would take hours upon hours to complete given the latencies involved with writing to CLARiiON’s earliest version of RAID5.

CLARiiON’s roadmap always planned to have a caching mechanism (similar to what Symmetrix had introduced years before). Adding the capability, however, was no small feat. In order to add write caching functionality, the engineering team would have to add battery-backup capability, a vaulting area to de-stage the cache in the case of a power failure, and an industry-first mirroring of cached data to a separate processor.

This historical use case is another example of how service level workloads drive innovation. Over time, many disk arrays would possess innovative features such as RAID5 and mirrored caching, but the earliest version were driven by new workloads.

In my next post I will introduce how customers began running diverse workloads against the storage infrastructure, which in turn introduced new advances into the industry.

image credit: wikimediacommons.org

Sign up here to get Human-Centered Change & Innovation Weekly delivered to your inbox every week.

GUEST POST from Steve Todd

In my last post I introduced the concept of application workload evolution. As software applications become increasingly complex, they drive innovation from the underlying IT infrastructure that has to service those applications.

In this post I’d like to dive into two specific examples. The first example describes application workloads in a high-end context. The second example describes a mid-range, open-systems context.

Cache Technology

In the 1980s, storage devices (such as HDAs) were limited in their ability to keep up with application workload demand. Hard drives might take 25-30 milliseconds to respond to an application request. Larger numbers of applications were deployed onto more and more server hosts (or mainframe environments).  The ability for the storage infrastructure to satisfy the application workload increase became impossible.

At the time, customers had only one solution: buy a new system (e.g. a new mainframe) with a new storage system attached. This was too expensive but often the only choice.

As a result, EMC engineers in Hopkinton, Massachusetts developed a cached disk array known as Symmetrix. A high-level diagram of a cached approach is shown below.

From the application point of view, the “disk drive” processed application requests with an order of magnitude more speed: (e.g. 2-3 milliseconds as opposed to 20-30). In some cases, the Symmetrix product removed the need for a customer to purchase a second system. The new capability to service I/O requests from a disk array cache allowed the customer to run more applications on their existing systems than ever before.

Up The Road in Westborough

Less than ten miles away from the Symmetrix engineers in Hopkinton, the Westborough-based Data General team was also struggling to solve the “slow disk drive” problem. However, they were targeting a different application workload (which resulted in a different innovation).

At the time, UNIX systems were on the rise and mid-range database systems were becoming in vogue. Operating systems such as Solaris were out in front from a device driver perspective, forcing more and more requests down onto the disk infrastructure. Database workloads could be characterized as having more reads than writes, often by a 2-1 ratio (or higher). A cached approach would likely be too expensive for customers in the mid-range; a RAID5 approach, however, would fit the bill nicely. With a random-read-weighted workload the
striping of data across multiple disks enabled IO parallelism. This innovation
sped up the storage response time for the database workload.

At the time, RAID5 was an academic concept that proved to be highly challenging to implement, due in large part to the enormous number of failure permutations that could occur during a RAID5 write operation (sometimes referred to as a write hole).

The resulting RAID5 product implementation, initially known as HADA, and eventually better-known as CLARiiON, effectively satisfied the workload demands of typical mid-range database applications.

Innovation via Application Workload

This brings me back to the main point of the post. In the 1980s, the evolution of application workloads drove high-tech innovation. The same is still true today.  The diagram below shows a small sample of the variety of workloads that must be satisfied by today’s IT infrastructures.

Most of the historical discussion up to this point has focused on the upper portion of the Y-axis (performance). In a future post, I’ll take a look at the X-axis (service level) and relate how the service level needs of an application also play a big factor in driving innovation.

image credit: nyse.com

Sign up here to get Human-Centered Change & Innovation Weekly delivered to your inbox every week.

GUEST POST from Steve Todd

A good way to frame any innovation discussion in high-tech is in the context of application workloads.

The slide shown below has become very common both inside and outside of EMC. It attempts to categorize the IT infrastructure needs of application workloads.

From an engineering standpoint, the Y-axis can be difficult to comprehend. The top portion of the Y-axis represents performance (often represented using units such as “I/Os per second”), while the bottom portion of the Y-axis represents capacity (e.g. often represented using units such as “terabytes”).  For an engineer used to the Cartesian Coordinate system, one would expect the units to be consistent for the entire Y-axis (not to mention the X-axis as well)!

It was exactly this discussion that EMC Distinguished Engineer Mark Lippitt and I were having last week. Mark has been involved in the storage industry since the late 1970s, and he and I were taking a historical look at what the chart is trying to convey.

Both of us agreed with the fundamental premise of the chart: application workload requirements drive innovation and dramatic change into existing IT infrastructures.

It is interesting to discuss the idea of application workload evolution in the context of one of the dominant information storage protocols emerging from the 1980s: the SCSI protocol. One of the first jobs I had after college graduation was a performance evaluation of workloads between SCSI and ESDI. During this time I learned about the capabilities of the SCSI protocol. In particular I learned about tagged command queueing, and began to understand, for the first time, that disk technology was not keeping up with application workloads.

In a recent post I wrote about the concept of application nearness. I used the illustration below to indicate that in the 1980s the applications were compiled to run on a CPU that was geographically and physically quite close to a spinning disk drive. One of the capabilities of SCSI command tagged queueing, as illustrated below, was the ability for the disk drive to accept more than one request at a time (e.g. the request to store the values “1”, “2”, “3”,  and “4” are all issued by the application before the first request is finished).

This picture is meant to highlight that the applications running on “fast” CPUs were spending a lot of time waiting for a hard disk drive to perform a series of mechanical movements to store the data.

In the context of the workload graph shown above, the application workloads of the 1980s were driving the performance requirements further up the Y-axis. At that point in history, an application would typically perform all of its read and write request to one hard disk drive, which could only handle (for example) the completion of one request every 20 milliseconds.

Interestingly enough, two alternative approaches, by two separate teams, in two separate companies, were developed less than 10 miles away from each other. Both of these approaches, which are still valid and operational 30 years later, ushered the industry into the disk array era. Note that the Wikipedia definition of disk array displays the products deploying these alternative approaches).

I will spend some time in an upcoming approach diving into how application workload played a role in driving these two innovations.

In the meantime, Mark and I agreed that while the Cartesian coordinate approach for describing application workload may annoy the engineer, it is a highly effective framework for starting a great dialogue.

image credits: tcc.edu; emc.com

Sign up here to get Human-Centered Change & Innovation Weekly delivered to your inbox every week.