Tag Archives: Experimentation

“I don’t know,” is a clue you’re doing it right

“I don’t know,” is a clue you’re doing it right

GUEST POST from Mike Shipulski

If you know how to do it, it’s because you’ve done it before. You may feel comfortable with your knowledge, but you shouldn’t. You should feel deeply uncomfortable with your comfort. You’re not trying hard enough, and your learning rate is zero.

Seek out “don’t know.”

If you don’t know how to do it, acknowledge you don’t know, and then go figure it out. Be afraid, but go figure it out. You’ll make mistakes, but without mistakes, there can be no learning.

No mistakes, no learning. That’s a rule.

If you’re getting pressure to do what you did last time because you’re good at it, well, you’re your own worst enemy. There may be good profits from a repeat performance, but there is no personal growth.

Why not find someone with “don’t know” mind and teach them?

Find someone worthy of your time and attention and teach them how. The company gets the profits, an important person gets a new skill, and you get the satisfaction of helping someone grow.

No learning, no growth. That’s a rule.

No teaching, no learning. That’s a rule, too.

If you know what to do, it’s because you have a static mindset. The world has changed, but you haven’t. You’re walking an old cowpath. It’s time to try something new.

Seek out “don’t know” mind.

If you don’t know what to do, it’s because you recognize that the old way won’t cut it. You know have a forcing function to follow. Follow your fear.

No fear, no growth. That’s a rule.

Embrace the “don’t know” mind. It will help you find and follow your fear. And don’t shun your fear because it’s a leading indicator of novelty, learning, and growth.

Image credit: Pixabay

Subscribe to Human-Centered Change & Innovation WeeklySign up here to join 17,000+ leaders getting Human-Centered Change & Innovation Weekly delivered to their inbox every week.

The Importance of Experimentation in Innovation

The Importance of Experimentation in Innovation

GUEST POST from Art Inteligencia

Innovation is crucial for the success and growth of any organization. However, many companies struggle to consistently come up with new and creative ideas that drive progress. This is where the importance of experimentation in innovation comes into play. Experimentation enables companies to test new ideas, learn from failures, and ultimately develop groundbreaking innovations.

Case Study 1: Amazon

One of the key benefits of experimentation in innovation is the ability to fail fast and fail cheap. By testing multiple ideas and approaches, companies can quickly identify what works and what doesn’t, reducing the risk of investing time and resources into projects that are unlikely to succeed. For example, Amazon’s product development process is driven by experimentation and continuous testing. The company encourages teams to take risks and experiment with new features and products, knowing that failure is a natural part of the innovation process. This approach has enabled Amazon to create game-changing products like Amazon Prime and the Kindle e-reader.

Case Study 2: Google

Another example of the importance of experimentation in innovation is the case of Google’s self-driving car project. Through a series of experiments and iterations, Google’s engineers were able to develop a fully autonomous vehicle that has the potential to revolutionize the transportation industry. The team behind the project embraced a culture of experimentation, constantly testing and refining their ideas to overcome technical challenges and improve the safety and performance of the vehicle. This commitment to experimentation has allowed Google to stay at the forefront of autonomous vehicle technology and drive innovation in the automotive sector.


Experimentation is a critical component of the innovation process. By testing new ideas, learning from failures, and continuously refining their approach, companies can drive meaningful innovation and stay ahead of the competition. Amazon and Google are just two examples of organizations that have leveraged experimentation to develop groundbreaking products and technologies. Embracing a culture of experimentation can give companies a competitive advantage and position them for long-term success in a rapidly changing business landscape.

Bottom line: Futurology is not fortune telling. Futurists use a scientific approach to create their deliverables, but a methodology and tools like those in FutureHacking™ can empower anyone to engage in futurology themselves.

Image credit: misterinnovation.com

Subscribe to Human-Centered Change & Innovation WeeklySign up here to get Human-Centered Change & Innovation Weekly delivered to your inbox every week.

The Surprising Power of Business Experiments

The Surprising Power of Business ExperimentsInterview with Stefan H. Thomke

I had the opportunity recently to interview fellow author Stefan H. Thomke, the William Barclay Harding Professor of Business Administration at Harvard Business School to talk with him about his new book Experimentation Works: The Surprising Power of Business Experiments, to explore the important role that experimentation plays in business and innovation.

1. Why is there a business experimentation imperative?

My book Experimentation Works is about how to continuously innovate through business experiments. Innovation is important because it drives profitable growth and creates shareholder value. But here is the dilemma: despite being awash in information coming from every direction, today’s managers operate in an uncertain world where they lack the right data to inform strategic and tactical decisions. Consequently, for better or worse, our actions tend to rely on experience, intuition, and beliefs. But this all too often doesn’t work. And all too often, we discover that ideas that are truly innovative go against our experience and assumptions, or the conventional wisdom. Whether it’s improving customer experiences, trying out new business models, or developing new products and services, even the most experienced managers are often wrong, whether they like it or not. The book introduces you to many of those people and their situations—and how business experiments raised their innovation game dramatically.

2. What makes a good business experiment, and what are some of the keys to successful experiment design?

In an ideal experiment, testers separate an independent variable (the presumed cause) from a dependent variable (the observed effect) while holding all other potential causes constant. They then manipulate the former to study changes in the latter. The manipulation, followed by careful observation and analysis, yields insight into the relationships between cause and effect, which ideally can be applied and tested in other settings. To obtain that kind of learning—and ensure that each experiment contains the right elements and yields better decisions—companies should ask themselves seven important questions: (1) Does the experiment have a testable hypothesis? (2) Have stakeholders made a commitment to abide by the results? (3) Is the experiment doable? (4) How can we ensure reliable results? (5) Do we understand cause and effect? (6) Have we gotten the most value out of the experiment? And finally, (7) Are experiments really driving our decisions? Although some of the questions seem obvious, many companies conduct tests without fully addressing them.

Here is a complete list of elements that you may find useful:


  • Is the hypothesis rooted in observations, insights, or data?
  • Does the experiment focus on a testable management action under consideration?
  • Does it have measurable variables, and can it be shown to be false?
  • What do people hope to learn from the experiments?


  • What specific changes would be made on the basis of the results?
  • How will the organization ensure that the results aren’t ignored?
  • How does the experiment fit into the organization’s overall learning agenda and strategic priorities?


  • Does the experiment have a testable prediction?
  • What is the required sample size? Note: The sample size will depend on the expected effect (for example, a 5 percent increase in sales).
  • Can the organization feasibly conduct the experiment at the test locations for the required duration?


  • What measures will be used to account for systemic bias, whether it’s conscious or unconscious?
  • Do the characteristics of the control group match those of the test group?
  • Can the experiment be conducted in either “blind” or “double-blind” fashion?
  • Have any remaining biases been eliminated through statistical analyses or other techniques?
  • Would others conducting the same test obtain similar results?


  • Did we capture all variables that might influence our metrics?
  • Can we link specific interventions to the observed effect?
  • What is the strength of the evidence? Correlations are merely suggestive of causality.
  • Are we comfortable taking action without evidence of causality?


  • Has the organization considered a targeted rollout—that is, one that takes into account a proposed initiative’s effect on different customers, markets, and segments—to concentrate investments in areas when the potential payback is the highest?
  • Has the organization implemented only the components of an initiative with the highest return on investment?
  • Does the organization have a better understanding of what variables are causing what effects?


  • Do we acknowledge that not every business decisions can or should be resolved by experiments? But everything that can be tested should be tested.
  • Are we using experimental evidence to add transparency to our decision-making process?

Experimentation Works3. Is there anything special about running online experiments?

In an A/B test, the experimenter sets up two experiences: the control (“A”) is usually the current system—considered the champion—and the treatment (“B”) is some modification that attempts to improve something—the challenger. Users are randomly assigned to the experiences, and key metrics are computed and compared. (A/B/C or A/B/n tests and multivariate tests, in contrast, assess more than one treatment or modifications of different variables at the same time.) Online, the modification could be a new feature, a change to the user interface (such as a new layout), a back-end change (such as an improvement to an algorithm that, say, recommends books at Amazon), or a different business model (such as an offer of free shipping). Whatever aspect of customer experiences companies care most about—be it sales, repeat usage, click-through rates, or time users spend on a site—they can use online A/B tests to learn how to optimize it. Any company that has at least a few thousand daily active users can conduct these tests. The ability to access large customer samples, to automatically collect huge amounts of data about user interactions on websites and apps, and to run concurrent experiments gives companies an unprecedented opportunity to evaluate many ideas quickly, with great precision, and at a negligible cost per additional experiment. Organizations can iterate rapidly, win fast, or fail fast and pivot. Indeed, product development itself is being transformed: all aspects of software—including user interfaces, security applications, and back-end changes—can now be subjected to A/B tests (technically, this is referred to as full stack experimentation).

4. What are some of the keys to building a culture of large-scale experimentation?

Shared behaviors, beliefs, and values (aka culture) are often an obstacle to running more experiments in companies. For every online experiment that succeeds, nearly 10 don’t—and in the eyes of many organizations that emphasize efficiency, predictability, and “winning,” those failures are wasteful. To successfully innovate, companies need to make experimentation an integral part of everyday life—even when budgets are tight. That means creating an environment in which employees’ curiosity is nurtured, data trumps opinion, anyone (not just people in R&D) can conduct or commission a test, all experiments are done ethically, and managers embrace a new model of leadership. More specifially, companies have addressed some of these obstacles in the following ways:

They Cultivate Curiosity

Everyone in the organization, from the leadership on down, needs to value surprises, despite the difficulty of assigning a dollar figure to them and the impossibility of predicting when and how often they’ll occur. When firms adopt this mindset, curiosity will prevail and people will see failures not as costly mistakes but as opportunities for learning. Many organizations are also too conservative about the nature and amount of experimentation. Overemphasizing the importance of successful experiments may inadvertently encourage employees to focus on familiar solutions or those that they already know will work and avoid testing ideas that they fear might fail.

They Insist That Data Trump Opinions

The empirical results of experiments must prevail when they clash with strong opinions, no matter whose opinions they are. But this is rare among most firms for an understandable reason: human nature. We tend to happily accept “good” results that confirm our biases but challenge and thoroughly investigate “bad” results that go against our assumptions. The remedy is to implement the changes experiments validate with few exceptions. Getting executives in the top ranks to abide by this rule is especially difficult. But it’s vital that they do: Nothing stalls innovation faster than a so-called HiPPO—highest-paid person’s opinion. Note that I’m not saying that all management decisions can or should be based on experiments. Some things are very difficult, if not impossible, to conduct tests on—for example, strategic calls on whether to acquire a company. But if everything that can be tested online is tested, experiments can become instrumental to management decisions and fuel healthy debates.

They Embrace a Different Leadership Model

If most decisions are made through experiments, what’s left for managers to do, beyond developing the company’s strategic direction and tackling big decisions such as which acquisitions to make? There are at least three things:
Set a grand challenge that can be broken into testable hypotheses and key performance metrics. Employees need to see how their experiments support an overall strategic goal.

Put in place systems, resources, and organizational designs that allow for large-scale experimentation. Scientifically testing nearly every idea requires infrastructure: instrumentation, data pipelines, and data scientists. Several third-party tools and services make it easy to try experiments, but to scale things up, senior leaders must tightly integrate the testing capability into company processes.

Be a role model. Leaders have to live by the same rules as everyone else and subject their own ideas to tests. Bosses ought to display intellectual humility and be unafraid to admit, “I don’t know…” They should heed the advice of Francis Bacon, the forefather of the scientific method: “If a man will begin with certainties, he shall end in doubts; but if he will be content to begin with doubts, he shall end in certainties.”

Continue reading the article on InnovationManagement.se

Accelerate your change and transformation success

Subscribe to Human-Centered Change & Innovation WeeklySign up here to get Human-Centered Change & Innovation Weekly delivered to your inbox every week.

How to Foster a Culture of Experimentation

Unlocking Innovation Potential

How to Foster a Culture of Experimentation

GUEST POST from Chateau G Pato

In today’s fast-paced and ever-changing business environment, innovation has become a vital aspect of success for organizations across industries. Companies must constantly explore new ideas, products, and processes to stay ahead of the competition. However, fostering a culture of experimentation within an organization can be challenging. It requires a mindset that embraces failure as a stepping stone to success and encourages employees to think outside the box. In this article, we will explore the importance of experimentation and highlight two case studies that demonstrate how organizations have successfully unlocked their innovation potential.

Case Study 1: Google’s 20% Time

Google is a pioneer in fostering a culture of experimentation through its well-known “20% time” policy. Starting in the early 2000s, Google allowed its employees to dedicate 20% of their workweek to pursue projects of their own choosing, even if those projects were unrelated to their current roles. This policy encouraged employees to think creatively, take risks, and work on innovative ideas that were not part of their daily responsibilities.

This culture of experimentation led to the creation of successful products like Gmail, Google Maps, and AdSense, which all began as side projects during employees’ 20% time. By giving employees the freedom to explore their passions and experiment with new ideas, Google was able to tap into the collective potential of its workforce, resulting in groundbreaking innovations.

The success of Google’s 20% time policy illustrates the power of fostering a culture that promotes experimentation and risk-taking within an organization. By providing employees with the space and autonomy to dedicate time to their own projects, companies can unlock new perspectives, drive creativity, and spark innovation.

Case Study 2: Amazon’s Fail Fast Culture

Another excellent example of fostering a culture of experimentation is demonstrated by Amazon. Amazon has a “fail fast” approach, which encourages employees to test out new ideas quickly, learn from failures, and iterate rapidly. This mindset emphasizes the importance of taking calculated risks and accepting that not all experiments will succeed.

One notable example is Amazon’s foray into the smartphone market with the launch of the Fire Phone in 2014. Despite heavy investments, the Fire Phone failed to gain traction in the market and faced significant backlash. Instead of dwelling on this failure, Amazon quickly learned from the experience, pivoted its strategy, and went on to introduce successful products like the Kindle Fire tablet and the Amazon Echo.

Amazon’s fail fast culture allowed the company to bounce back from setbacks and leverage the knowledge gained through experimentation to drive future successes. By fostering a culture that embraces failure as a valuable learning experience, Amazon encourages its employees to take risks and explore new possibilities, spurring innovation throughout the organization.


Unlocking innovation potential and fostering a culture of experimentation is crucial for organizations looking to stay competitive in today’s dynamic business landscape. By learning from real-life case studies like Google’s 20% time policy and Amazon’s fail fast culture, businesses can gain insights into how to create an environment that encourages creativity, risk-taking, and continuous learning.

To foster a culture of experimentation, organizations should empower employees with autonomy, provide dedicated time for innovative projects, and foster an environment where failures are seen as learning opportunities rather than obstacles. By embracing experimentation and cultivating a mindset that values and encourages innovation, organizations can unlock their full potential and drive sustainable growth in the long run.

Bottom line: Futurology is not fortune telling. Futurists use a scientific approach to create their deliverables, but a methodology and tools like those in FutureHacking™ can empower anyone to engage in futurology themselves.

And to help you with your culture of experimentation, please be sure to download Braden Kelley’s FREE Experiment Canvas, which you can print as a 35″x56″ poster or an 11″x17″ or use as a background in online whiteboarding tools like Miro, Mural, Lucidspark, Google Jamboard and Microsoft Whiteboard.

Image credit: Misterinnovation.com

Subscribe to Human-Centered Change & Innovation WeeklySign up here to get Human-Centered Change & Innovation Weekly delivered to your inbox every week.

Innovation Through Experimentation

Strategies for Rapid Iteration

Innovation Through Experimentation

GUEST POST from Chateau G Pato

In today’s fast-paced and constantly evolving business landscape, innovation is the key to staying ahead of the competition. However, traditional approaches to innovation may not be enough to keep up with rapidly changing customer needs and preferences. To foster innovation, organizations must embrace a culture of experimentation and adopt strategies for rapid iteration. In this article, we will explore the importance of experimentation in driving innovation and discuss two case study examples to illustrate successful implementation.

Case Study 1: Google’s “20% Time”

One of the most famous examples of fostering innovation through experimentation is Google’s “20% time.” This initiative allows employees to spend 20% of their workweek, or one day, working on projects that interest them outside of their core responsibilities. This flexible structure encourages employees to explore new ideas and experiment with innovative solutions.

One notable outcome of Google’s 20% time is the creation of Gmail. Originally developed as an experiment by a Google engineer, the project emerged from the employee’s personal interest in improving email communication. Through rapid iteration and continuous experimentation, Gmail was refined and eventually launched as one of Google’s most successful products. This case study demonstrates how giving employees the freedom to experiment can lead to significant innovation and long-term success.

Case Study 2: Amazon’s A/B Testing

Amazon, the e-commerce giant, is renowned for its customer-centric approach and its relentless pursuit of innovation. One of the strategies Amazon uses to continuously iterate and improve its offerings is A/B testing. By testing different variations of a webpage, product listing, or feature, Amazon gathers quantitative data to make informed decisions about which version performs better. This data-driven approach allows them to quickly adapt and optimize their offerings to meet customer expectations.

An example of Amazon’s A/B testing is its product recommendation engine. By experimenting with different algorithms and design variations, Amazon continuously refines its recommendation engine to provide highly personalized and relevant product suggestions. This iterative process has played a significant role in enhancing the customer experience, boosting sales, and establishing Amazon as an industry leader.

Key Strategies for Rapid Iteration

1. Embrace Failure as Learning: Encourage a culture where failure is seen as an opportunity to learn and improve. Failure should not be punished but celebrated as a stepping stone towards success. By fostering an environment that values experimentation and risk-taking, organizations can encourage employees to think creatively and push boundaries.

2. Establish Rapid Feedback Loops: Implement processes that allow for quick feedback and iteration. Regularly gather feedback from customers, employees, and other stakeholders to identify areas for improvement. This feedback loop enables organizations to make iterative changes based on real-world data and inputs, leading to more relevant and effective solutions.

3. Set Clear Goals and Metrics: Clearly define innovation goals and establish measurable metrics to track progress. By setting concrete objectives, organizations can evaluate the success of their experiments and measure the impact on key performance indicators. This data-driven approach helps focus efforts on what truly matters and ensures that innovation initiatives align with overall business objectives.


Innovation through experimentation is crucial for organizations aiming to thrive in today’s rapidly changing business landscape. By adopting strategies for rapid iteration, businesses can foster a culture that encourages and celebrates innovation. The case study examples of Google’s “20% time” and Amazon’s A/B testing demonstrate how organizations can drive significant innovation by allowing employees to experiment and by leveraging quantitative data to inform decision-making. By embracing failure, establishing feedback loops, and setting clear goals and metrics, organizations can unleash their creative potential, adapt to evolving market dynamics, and stay ahead of the competition.

EDITOR’S NOTE: Braden Kelley’s Experiment Canvas™ can be a super useful FREE tool for your innovation or human-centered design pursuits.

“The Experiment Canvas™ is designed to help people instrument for learning fast in iterative new product development (NPD) or service development activities. The canvas will help you create new innovation possibilities in a more visual and collaborative way for greater alignment, accountability, and more successful outcomes.”

Image credit: misterinnovation.com

Subscribe to Human-Centered Change & Innovation WeeklySign up here to get Human-Centered Change & Innovation Weekly delivered to your inbox every week.