GUEST POST from Steve Todd

In my last post I mentioned that idea contest analytics can assist the spread of local university research being conducted in a country like China.  Social network graphs that model idea submission amongst distributed teams can actually build “knowledge pathways” that are ripe for the transferring of university research knowledge. The odds of knowledge transfer coming out of China increase with the decision by Chinese researchers to collaborate across borders with their international counterparts.

In this post I’d like to flip the example and describe how university research can assist any global employee in the process of idea generation and improvement.

Earlier this year I wrote a post announcing the kickoff of Innovation Roadmap 2013 (EMC’s yearly idea contest). I mentioned that an “Improve Idea” button had been added to our internal idea submission portal. This button allows idea submitters to correlate their still-forming ideas against thousands of previous ideas submitted by employees. The improve button is shown below.

Consider this process in the context of university research. Does an employee idea overlap with relevant university research?  Can they find and locate the global EMC employee that is in contact with that university? Can they discover the names of relevant faculty and students and read their publications?  Answering yes to any and all of these
questions can increase the quality of idea submission (which is the goal).

Consider the topic model presented in a previous post.  Topic modeling provides the ability to process the entirety of a university research portfolio and “bucketize” it into themes like “Cloud” and “Big Data”. The output of a topic modeling exercise is shown below.

It’s entirely possible for an employee to type in the text of their
idea, and have the Topic Modeling Toolbox analyze the idea and correlate it to the most appropriate bucket.  Within this bucket, the toolbox will also highlight which university research initiatives are likely “most relevant”. The diagram below symbolically represents this process.

It is common to envision a university research program in which a specific set of engineers is targeted for knowledge transfer. For example, within EMC it makes perfect sense for the research we conduct with the University of Limerick (Flash technology) gets “pushed” to EMC internal groups that might benefit from the results (e.g. the XtremIO all-Flash disk array team, or XtremSF server flash team).

How does the knowledge transfer turn into a corporate asset?  How can the research and the result be tied together?  The approach that EMC is considering is lineage-based. Read my previous blog post on this topic for more detail.

image credit: www.contrib.andrew.cmu.edu

Sign up here to get Human-Centered Change & Innovation Weekly delivered to your inbox every week.

GUEST POST from Steve Todd

In previous posts I explained how EMC leverages an analytic process to manage university research globally. Truth be told, the analytic process actually measures global innovation; university research is a subset of corporate innovation activities.  The context of university research is worth exploring on its own, however, and I took the opportunity to share our approach with partner faculty and students at EMC’s 2^nd Annual University Day.

Analyzing university research side-by-side with other corporate innovation activities has its advantages. In my last post, I shared specific data about a list of Chinese researchers that are actively involved in local university research.  The pie chart below highlights the set of researchers that collaborated with Chinese universities for a specific time period.

In addition to participating in local university research, the engineers at EMC Labs China are also actively involved in the yearly EMC idea contest known as the Innovation Roadmap. Historical idea submissions from global employees (8,000+ ideas and growing) are also stored in this innovation database.

Employees are encouraged to submit their ideas as “teams”. In fact, diversity of team members, whether it be geographic or by function, is highly encouraged. This diversity is a leading cause of increased idea quality, as I’ve discussed in a recent post.  As a result of storing many years’ worth of historical idea submissions, we can begin to visualize team submissions using social network analysis.  For example, the social graph below focuses on Chinese employees that continually surface as “strong collaborators” in the area of idea submission.

In the graph above it is apparent that Jidong Chen, for example, has a network of collaborators with which he submits ideas.  Chances are good that Jidong, during discussions with his peers, is sharing the work of his university counterparts either directly or indirectly.

A more important question, in my mind, is whether Jidong is sharing knowledge across boundaries. These boundaries could be geographic, they could be technology related (e.g. security researchers, compression researchers, etc), or they could be by function (collaborating with marketing, HR, finance, etc). In order to validate that any given EMC employee is indeed crossing boundaries in the transfer of university research knowledge, the analysis was run again with color-coded values representing country of origin.
The graph below is a zoomed-out picture of the same chart, with color coding of
each individual by geography.

The yellow dots represent Chinese employees. The yellow circles with red boundaries correspond to the red circles in the previous chart. One can readily see that Chinese idea submitters not only cluster together, but they bridge to other geographies as well. The colors of these circles represent employees in France, Israel, Australia, and the United States.  As a result, certain Chinese employees have extremely high betweenness ranks, which means that they are strong candidates to transfer their knowledge to other countries.

Whether or not they actually DO transfer that knowledge is a different thing altogether. Is it enough to know that the potential is there?  The answer is no. However, the knowledge of betweenness allows a centralized innovation program to guide good behaviors when it comes to global knowledge transfer. The analytic results give us all the pieces to put a plan into action.

This post has highlighted the ability to explore connectedness and collaboration emanating from an EMC employee conducting local university research. Knowledge expands locally, and then it has the potential to be transferred globally.

What about the reverse of this process? Can any global researcher correlate their research efforts to a remote employee like Jidong?

In a future post I will examine analytic techniques to enable this behavior.

image credit: coupeweb.ca

Sign up here to get Human-Centered Change & Innovation Weekly delivered to your inbox every week.

GUEST POST from Steve Todd

In a previous post I introduced an interesting new research project that my company (EMC) is working on with NUI Galway and Babson. During that post I quoted Professor Eoin Whelan‘s thoughts on Structural Holes:

Ron Burt’s theory of structural holes has proven to be influential in explaining how innovation transpires.  Burt proposes that gaps in a social network, structural holes, create brokerage opportunities.  A structural hole indicates that the people on either side of the hole circulate in different flows of information and advantages accrue to those individuals whose relationships span the structural hole.

In today’s digitally connected world, many social relationships are formed and maintained through social media.  The reason for the academic partnership is to examine if innovation theories such as Burt’s structural holes apply to networks formed through social media (and specifically Twitter).  The researchers are using social network analysis techniques to find correlations between thousands of employee ideas submitted into our Innovation Roadmap framework, and the structure of the submitter Twitter networks. The analysis of this data continues but initial findings suggest that the structural holes theory also applies to Twitter.  People with more disconnected twitter networks tend to submit better ideas, as determined by the average number of positive votes received by peers per idea.

I ask Eoin to provide some graphics that highlight their work so far, and he supplied two “sociograms”, which are described as follows:

The results can be better explained by examining the Twitter ‘sociograms’ of two EMC employees below.  Even though both employees roughly follow the same number of Twitter handles, employee A’s network is far more diverse.  In other words, the people employee A follows on Twitter are for the most part not following each other.  Mathematically, we can determine this level of diversity using the fragmentation ratio which for employee A is 87%.  The data is showing that highly fragmented Twitter networks, like employee A’s, are better for ideation.  In contrast, employee B’s Twitter network is quite compact.  Those
in employee B’s network are nearly all following each other, hence a low fragmentation rate of 12%.  Such cohesive networks provide more redundant information, which the data shows is negatively correlated with ideation.

In reviewing these preliminary research results inside EMC, Distinguished Engineer John Cardente pointed out that the structural holes concept is closely related to the Betweenness network attribute that we have socialized previously. Specifically, network nodes that span structural holes tend to have high betweenness scores.

I’ll continue to publish more of these results as the research moves forward.

image credits: steve todd; en.wikipedia.org

Sign up here to get Human-Centered Change & Innovation Weekly delivered to your inbox every week.

GUEST POST from Steve Todd

Several weeks ago I published a blog post (Different Minds Think Greatly) that explored the topic of cognitive diversity and innovation. At the time, I had read a Techonomy article by John Hagel and John Seely Brown, who basically asserted that “too much like-mindedness hurts companies”, and I quoted the following:

Organizations that host a diverse and broad range of members have a resilience that results from cross pollination.

As part of the article I echoed my agreement with this assertion and referred to some  Social Network Analysis from my own company (EMC). The data that we modeled  highlighted that a good degree of geographic diversity can result in higher quality ideas.  “Higher quality ideas” are typically defined as ideas that receive a high score from judges in our yearly Innovation Roadmap program (especially ideas that reach finalist or funded status).

Our data shows that when diverse minds from different cultures collaborate on new approaches, good ideas result.  The conclusions we drew from this analysis have resulted in behavioral change at EMC. Most notably we’ve formed a global “Innovation Best Practices Community” in order to intentionally stimulate this behavior.

After publishing some of our findings, we were approached by two universities on an interesting joint research project. They asked if we wouldn’t mind sharing a filtered view of our employee  idea activity in order to correlate it against the public Twitter profiles of these same people. Professors Eion Whelan (NUI Galway) and Salvatore Parise (Babson) invited us to focus on an offshoot of cognitive diversity known as “structural holes”. Eion explains structural holes in the following manner.

Ron Burt’s theory of structural holes has proven to be influential in explaining how innovation transpires.  Burt proposes that gaps in a social network, structural holes, create brokerage opportunities.  A structural hole indicates that the people on either side of the hole circulate in different flows of information and advantages accrue to those individuals whose relationships span the structural hole.  In his best selling book The Tipping Point, Malcolm Gladwell argues that the success of Paul Revere’s midnight ride was due to his quite diverse social networks – ranging from hunting and card playing to theatre and business. Therefore, he knew which doors to knock on when arriving in a town.  Network brokers like Revere not only disseminate information more broadly, they also benefit by receiving a greater novelty of information from their diverse social contacts.  Indeed, studies within organizations have shown that employees, teams, and even entire companies with more diverse network connections tend to be more innovative.

As a result of our conversation, we polled our global Innovation Best Practices community and asked idea submitters to voluntarily share their Twitter handles with Eoin and Sal. We packaged up the Twitter handles with the employees’ corresponding level of innovation activity over a period of several years.

This research has been ongoing for several months, and in an upcoming post I plan to share some of the results and what it might mean for our organization.  Before I do, however, I’d like to discuss this Data Science project in the context of Phase 1 of the Data Analytics Life Cycle: Hypothesis Generation.

Our hypothesis could be stated as follows:

EMC employees with diverse external Twitter networks submit higher quality ideas.

By “diverse” we mean “disconnected” or “fragmented”. In other words, employees that  follow people that are not “like-minded” tend to submit better ideas due to the diversity of their network.

If this hypothesis proves to be true, we can brainstorm ways of stimulating additional behavioral change via encouraging our employees to fragment their Twitter networks. Proving the hypothesis involves iterating through the additional phases of the Life Cycle (e.g. Data Prep, Data Modeling, etc).

In future posts I will share the results of the modeling exercise conducted by Eoin and Sal.

image credit: apple.com; minimal wall

Sign up here to get Human-Centered Change & Innovation Weekly delivered to your inbox every week.

GUEST POST from Steve Todd

Today I attended EMC’s 2^nd Annual University Day in Santa Clara, California.  A large number of schools were represented from all over the United States, including:

• UC San Diego
• UC Irvine
• UC Santa Cruz
• Northeastern University
• Minnesota
• Carnegie Mellon University
• University of Wisconsin
• Case Western
• Florida International University
• University of Utah
• Harvard
• University of Rochester
• Stony Brook University
• Princeton University

The agenda for the day included discussions on challenging high-tech issues in

next generation data centers, including new developments in solid state storage. EMC Distinguished Engineer Jeroen VanRotterdam led an interesting dialogue examining the current state of relationships between Industry and Academia.

Greg Ganger, CMU Professor and Director of the Parallel Data Lab, gave the Academic

Keynote during the afternoon session. His keynote was followed by the annual poster session, in which nine students competed for first prize.

For this post, however, I’d like to summarize a discussion I led just before lunch, in

which I asked the students the following question:

“How would you manage EMC’s global university research portfolio?”

Their answer was loud and clear: “We don’t know!” I responded that the answer was a fair one; it’s a hard problem to solve. I then shared our company’s approach of using EMC’s own analytic products (e.g. Pivotal/Greenplum) to perform global analytics across all academic research partners.

In order to highlight the global span and scope of our research initiatives, I shared the following map:

This map is dynamically generated. While it doesn’t represent every university research partnership EMC has across the globe, it’s pretty close.  The map is the result of nearly two years of collaboration across all of the countries that register their research engagements. The larger the circle, the more activity is being reported from the region.

What types of analysis can be run against a database containing research activities? During my talk I described the current reports enabled by our analytics framework:

• A visualization of the “types” of research currently active in our portfolio (e.g. solid state storage, analytics, etc).
• A visualization of the “types” of research by region (e.g. where in the world do we research compression technology?)
• Who are EMC’s key researchers in any given region?
• Which researchers are the best at transferring knowledge out of their region?
• For any given EMC researcher, what type(s) of research do they conduct?
• What is the complete list of EMC employees, per region, that are involved in any form of university research?
• How can global EMC employees advance their own ideas by locating relevant university research?
• How do we augment university research with other external employee connections (e.g. programmatically leverage their Twitter connections)

The talk was well-received. The faculty and students that attended got a good feel for the framework that EMC uses to impact our own business by expanding our knowledge with local university partners.

In future posts I will dive in many of the items above in more detail to specifically describe how analytics are leveraged to improve EMC’s university research results.

image credit: idc.com

Sign up here to get Human-Centered Change & Innovation Weekly delivered to your inbox every week.

GUEST POST from Steve Todd

One of the next waves of innovation in building data centers is related to the increasing number of layers between application and storage.

I’ve thought about this phenomena in the context of IDC’s Third Platform, which I describe briefly here:

• 1st Platform: Mainframe, Mini Computer, Terminals. Millions of users, thousands of apps.
• 2nd Platform: LAN/Internet, Client/Server. Hundreds of millions of users, tens of thousands of apps.
• 3rd Platform: Mobile, Cloud, Big Data, Social on Mobile Devices. Billions of users, millions of apps.

My career started at the end of the mainframe era. I have a graphic which I refer to as “pre-RAID” storage architecture, and this diagram can be used to describe the “nearness” of the application to the storage itself. This picture shows a CPU sending values directly to a disk drive.

The application, in this example, would run directly on the CPU.

At the beginning of the client/server (2nd platform) era, the application begin to transition “further away” from the disk in terms of virtualized layers. The diagram below introduces a virtualized RAID layer (not shown) and a write cache above the disk drives. One can begin to graphically see the application move “further” away from the application storage repository.

As we fast forward to the ending of the client/server era and the beginning of the 3rd platform era, we now see virtualization at every level, and an increased amount of “distance” (in terms of layers) between the storage and the application.  My EMC colleague Ken Durazzo likes to use the following diagram to depict an examplar data center application stack.

Application nearness is clearly decreasing, and as the vision of the 3rd platform advances, applications will move even further away (think mobile apps connecting into the stack depicted above).

The coordination of all of the plumbing and wiring between the application and storage is where the bulk of innovation will surface in the next few years.  Cloud management platforms (like OpenStack) and Network Virtualization technologies (like SDN) are getting a lot of buzz right now for just this reason.

Data center architectures are transforming significantly, but legacy configurations can’t flip over to radically different paradigms. DC architects are juggling a lot of balls in the air right now (see Doing Three Things at Once).

In future posts I hope to discuss different architectural approaches to cope with the application nearness phenomena.

image credit: idc.com

Sign up here to get Human-Centered Change & Innovation Weekly delivered to your inbox every week.

GUEST POST from Steve Todd

Earlier this week the Chief Technology Officer of EMC, John Roese, launched his own version of what used to be called the Innovation Showcase:  The 2013 Innovation Roadmap.

The Innovation Showcase was launched in 2007, and for six consecutive years the ideation growth across EMC’s global employee base has been consistent.  In 2007 employees submitted just over 400 ideas; in 2012 they submitted a grand total of 2200 ideas. Each idea was targeted at a specific challenge that EMC is facing as a company. In many cases these challenges come directly from EMC Executives that wish to extend beyond their organization for a broader set of suggestions to the problems that they face. Every year this process has been celebrated as an “idea showcase” at EMC’s annual (and global) Innovation Conference.

This year John has renamed the program to build upon the ideation success of the previous six years and focus on the continual delivery of employee ideas as a scheduled, predictable roadmap. As part of this effort the Office of the CTO has asked all business units to also financially sponsor the Innovation Roadmap at a Bronze, Silver, Gold, or Platinum level. This fund can be leveraged during the incubation stage.

Big Data for Innovation Insight

One of the benefits of being a Big Data company is the leverage of our own analytic technologies.  For several years I have been describing our use of Greenplum technology for the purpose of creating a global innovation analytics sandbox. Several graduates of EMC’s Data Scientist course (including myself) have taken our learnings and applied analytics on this global innovation data.

The visualizations have influenced our corporate innovation decisions, and perhaps the best example of this is the “Improve Idea” button that our employees can press before submitting their idea to EMC’s 60,000+ person employee base.

Below are some of the innovation analytics that led to the creation of the Improve Idea button:

• We have been running a longitudinal study of idea incubation progress. This study allows us to measure collaboration (or lack of collaboration) that moves (or doesn’t move) ideas forward.
• We have measured the evolution of idea submission trends over time, and recognized that our employees do not have an easy way to explore previous ideas and how they relate to new ideas.
• We’ve created global social network graphs that highlight idea collaboration and idea quality. We’ve also identified boundary spanners (ideators whose influence spans countries).
• We identified clusters of innovators and understood the culture that made them successful.
• We analyzed word count on idea submission and informed our employees that more well-thought out and articulated ideas have a higher success ratio.
• We discovered that successful ideation programs in a given region quite often have a robust innovation framework in place.
• We studied clique size and the impact on idea effectiveness.

EMC’s Data Scientist curriculum teaches that the last two phases of the Data Analytic Life Cycle (Communicate Results and Operationalize) should have a goal of driving behavioral change. Our analytics supplied a wealth of evidence that idea collaboration strengthens idea quality. We therefore put a plan in place to encourage collaboration by connecting inventors before they actually submit their ideas.

The button shown below is the result of that effort. I will dive into our implementation of the Improve Button in a future post.

In the meantime, I encourage my EMC co-workers to “push it”. The Innovation Central submission server is open until July 12!

image credit: emc.com

Sign up here to get Human-Centered Change & Innovation Weekly delivered to your inbox every week.

GUEST POST from Steve Todd

In March I visited two universities in Egypt that have deployed “mini-clouds” for the purpose of education and innovation: German University Cairo (GUC) and Ain Shams University.

This visit is related with a blog post I wrote several years ago (2011) called Cairo Cloud Formations. In that post, I discussed how the Egyptian innovation ecosystem was “primed and ready” to learn about cloud computing, design cloud architectures, build their own systems, and innovate within those systems.

Looking back on that 2011 visit, several things come to mind:

• Cloud computing was a government priority. I met with the Ministry of Communications and Information Technology (MCIT) and they had already adopted EMC’s Cloud Computing Curriculum to bring them up to speed on the topic.
• The university research community in Cairo had multiple professors that were well-versed in cloud computing, but the student population as a whole were not aware of the cloud computing definition, benefits, and approaches.
• My EMC co-workers were well aware of EMC’s cloud computing strategy, but none of them had hands-on experience with physically and logically piecing together a cloud computing infrastructure.

Our strategy after this visit was to (a) focus heavily on cloud education, (b) generate cloud-specific research proposals with select universities, and (c) build a “mini-cloud” on-premise at the Cairo Center of Excellence with EMC employees.

Two years later, there has been significant progress made on all three fronts.

The “mini-cloud” action item, however, has had the most impact in terms of advancing the state of cloud computing knowledge (and practice).

The mini-cloud was designed by EMC Distinguished Engineer Wissam Halabi.  To say that Wissam has cloud knowledge would be an understatement.  Wissam is EMC’s leading Cloud Architect and a major force behind the implementation of EMC’s internal private cloud, which supports over 60,000 employees, enables over 400,000 customer/partners, spans 5 geographic data centers, contains over 8 PB of data, and hosts well over 400 unique applications and tools.

So we asked Wissam: what is the minimum configuration that we could build in Egypt that qualifies as a “cloud”?

Below is a picture of his mini-cloud architecture:

It is not my purpose to step through this architecture, explain the components, and articulate how this minimalistic architecture satisfies certain cloud computing characteristics (although I can certainly work with Wissam on a separate blog post if there is interest).

Instead I prefer to focus on the impact this approach had on the innovation ecosystem. EMC employees were able to augment their cloud knowledge with a hands-on activity. They communicated this new knowledge to relevant government and university partners. Customers were brought in to the lab and educated on the approach. Without exception, our government and university partners raised their hands and said “we would like to do the same thing”.

As a result we have launched two EMC cloud computing research labs. These labs are dedicated to serve as a sandbox for (a) students completing the EMC Academic Alliance cloud computing course, and (b) researchers desiring a location to try out new cloud algorithms and ideas. Each lab consists of a VNXe, several servers, and VMware cloud assets.

Why VNXe?  Because the system in and of itself is cloud-like.

The mini-cloud approach is cookie cutter; any university can build a similar system for students.

In upcoming posts I will look at some of the research emerging from these systems, and share a bit more detail about EMC Cairo’s advanced innovation program.

GM Magued Mahmoud, myself, Innovation and University Research Program Manager Marwa Zaghow, and Distinguished Engineer Wissam Halabi with the “mini-cloud” at German University Cairo.

image credit: emc.com

Sign up here to get Human-Centered Change & Innovation Weekly delivered to your inbox every week.

GUEST POST from Steve Todd

In my recent post How to Get the Wrong Answer, I stated that analytic correctness (e.g. the “right answer”) increases when large volumes of data are highly varied.

Analytic correctness also has a corollary statement: the right answer is wrong if it takes too long too calculate.

A good example of this corollary can be found in high-tech security. Consider the chart below, which depicts the evolution of an intrusion, and the criticality of reducing the amount of time the intrusion goes undetected:

This slide was adapted from a report jointly created by the North American Electric Reliability Corporation and the U.S. Department of Energy. The report describes how sophisticated intruders are creating Advanced Persistent Threats (APTs) that penetrate and persist within corporate firewalls (page 32):

Advanced Persistent Threats (APT) are becoming a significant concern across all sectors. These threats involve sophisticated, determined, coordinated attackers who systematically compromise government and commercial computer networks. These attackers typically install multiple backdoors into a cyber network they are attempting to infiltrate, under the “radar” of even the most sophisticated anti-virus protections, thereby establishing a secure foothold into the network. They then install utilities to exfiltrate data to external servers. Attackers respond to attempts to eradicate infection and remediate network security by establishing additional footholds and improving sophistication. These infiltrations can persist, untraced, for months and even years.

Given this backdrop, one obvious strategy is to identify and address intrusions as quickly as possible. Many corporations rely on SIEM technologies and solutions(Security Information and Event Management) to accomplish this objective. These solutions traditionally provide real-time analysis of security alerts in order to detect intrusions.

Some of these approaches, however, are running out of gas. Traditional SIEM analytic tools no longer supply the right answer fast enough. Why?  Because the old approach lacks variety. The old approach relies too much on one form of data: security logs. Log-heavy analytics run the risk of either missing the intrusion or taking too long to identify it. IT operators in a Security Operations Center (SOC) may try and augment the security logs with other forms of input, but their SIEM infrastructures either weren’t designed for that volume of data, or they weren’t designed to handle massive varieties of streaming data.

A re-think of SIEM infrastructure was required and a new, innovative approach became necessary.

Chuck Hollis did a great job summarizing a new approach for security analytics in the data center. The model capitalizes on new data center architectures that I have written about previously: the ability to analyze massive amounts of recent, streaming security data alongside of a deep historical archive.

Instead of a log-heavy approach, this new style of SIEM architecture accepts a massive variety and volume of data. In addition to traditional security logs, these new repositories also contain complete firewall data,
network configurations, operating system state, a complete list of data center assets (e.g. a CMDB or configuration management data base), and network traffic traces.

The system requirements for keeping track of this variety of data will challenge a SIEM architecture designed for fewer sources of information. Capturing all network traffic, for example, is a wildly different use case than collecting security logs from devices.

The new approach espouses massive variety. Analytics running on top of this amount of variety have a much better chance of distilling out indications of attacks, threats, and vulnerabilities. Analytic models that run on top of these new architectures can take into account streaming and real-time data, as well as deep historical data.

Converged data center architectures, in which customers choose to buy pre-packaged cloud infrastructure (e.g. VCE), become a more secure choice. All components in the converged platform can pre-integrate to work in unison and cooperation with this new type of security analytics platform.

Does this mean that data center architectures that “mix and match” cloud infrastructure components are inherently less secure? Not necessarily. There has been recent innovation in this area that I will explore in a future post.

image credit: emc.com

Sign up here to get Human-Centered Change & Innovation Weekly delivered to your inbox every week.

GUEST POST from Steve Todd

In the 1980s, the introduction of RAID technology set off a wave of innovation in high-tech. This innovation was in response to a very simple customer request: “Never give me the wrong answer”.

RAID technology, in a nutshell, uses math to rebuild missing data (e.g. data that was only present on a disk drive that has since failed). I often use the diagram below to depict a scenario where data from a failed disk is at risk. The missing disk should contain the value “3”. Mathematical algorithms can recover the value “3” by subtracting the remaining values from the number “10”.

Operating system and server vendors argued that the math should run at the server level. They reasoned that the CPU is the brain, and the brain has the smarts to solve the problem.  Storage geeks argued otherwise; the failure permutations were so complex that the entire mathematical exercise needed to be offloaded to a storage CPU.

The storage geeks, in the end, were right. Ultimately the disk array industry was born.

The disk array approach promised (and delivered) “correctness”. Hospitals, governments, banks, and big business all relied on the mathematics running closer to the disks. If the data was ever delivered incorrectly (even just once!), the results could be tragic. Running the math in the server increased the likelihood of incorrectness.

A New Kind of Correctness

In 2013 I am seeing a very similar customer request, a very similar need for mathematical algorithms,  a very similar architectural dilemma between server and storage, and an eventual solution which will drive innovation in much the same way!

Customers are no longer asking for the “bit-for-bit” correctness of the 1980s (they assume this problem has already been solved).

They are asking for “analytic correctness”, which is a very different thing indeed. They point their mathematical algorithms at vast amounts of data and wait for the correct answer.

Where should customers run these analytic algorithms?  Shouldn’t they run on lighting-fast server CPUs?

Wrong answer!

The math should (once again) shift into the storage system. The analytic result will be more correct by running the analytic models closer to the data.

Correct analytic results are a function of processing a massive variety (and massive amount) of input sources. The more variety the better. The more volume the better. A traditional, CPU-heavy DBMS architecture can’t give you that. Here’s why:

• A traditional DBMS does not have nearly enough scale-out ingest ports for incoming data. As a result, the analytic models have access to less data, and the data is often “older”.
• There is too much data to be dragged out of traditional storage systems and brought up to the mathematical algorithms at the CPU level. Getting the correct answer takes longer.
• The variety of incoming data is structured and unstructured; these incoming streams get partitioned and add additional burden on the CPU to sort them out during modeling.

So what kind of innovation is needed to satisfy this new form of correctness?

In a nutshell, we need scale-out, shared-nothing storage systems that can ingest massive amounts of data and run the customer analytics in a highly-parallel fashion, right alongside the data!

I’ll be spending some time in 2013 diving down a bit further into the specifics of this approach (e.g. how to get the right answer).

From an innovation standpoint, EMC and VMware plan on capitalizing on this industry shift via the recently-announced spin-off: the Pivotal Initiative. As more and more customers look to leverage analytic capabilities in a cloud environment (and as cloud providers begin to provide those services), the Pivotal Initiative plans on building the enabling framework to make it happen.

Don’t get the wrong answer. Run the analytics closer to the storage and leverage both variety and volume.

image credit: www.abc.net.au

Sign up here to get Human-Centered Change & Innovation Weekly delivered to your inbox every week.