Evidence-Based Management — Certified Course

Chapter 1:

The Basic Principles

• Summarize the basic principles of evidence-based management;
• Explain why we need evidence-based management;
• Explain what counts as evidence;
• Determine which sources of evidence were consulted;
• Assess (coarsely and in general terms) the quality of evidence;
• Determine whether the ‘best available’ evidence was used in a decision-making process;
• Correct common misconceptions about evidence-based management.

Chapter 2:

Ask

• Identify (hidden) claims/assumptions regarding a practical issue;
• Determine whether an (assumed) problem is sufficiently clear;
• Determining whether there is sufficient evidence to support the (assumed) problem;
• Determine whether the preferred solution is sufficiently clear;
• Determine whether there is sufficient evidence (from multiple sources) to support the preferred solution.

Chapter 3:

ACQUIRE – evidence from practitioners

• Determine what evidence to acquire from practitioners;
• Determine how to prevent selection bias when acquiring evidence from practitioners;
• Determine the best method(s) to acquire evidence from practitioners;
• Determine whether bias could have affected evidence from practitioners;
• Formulate clear, unambiguous, and unbiased questions.

Chapter 4:

APPRAISE – evidence from practitioners

• Assess whether professional experience is valid and reliable;
• Grade the trustworthiness of professional experience;
• Recognize how system 1 thinking influences valid and reliable professional expertise;
• Determine whether a decision is based on system 1 or system 2 thinking;
• Recognize common cognitive biases;
• Identify ways cognitive biases can be overcome;
• Critically appraise evidence from practitioners.

Chapter 5:

ACQUIRE – scientific evidence

• Determine the most relevant online research database(s) given the question;
• Determine whether a journal is peer reviewed; Determine the most important PICOC terms;
• Search the Internet for relevant alternative and/or related terms;
• Search Google Scholar for related or broader academic terms;
• Test search terms to identify terms that yield the most relevant results;
• Apply Boolean operators to specify a search query;
• Use the history function to combine search queries;
• Apply methodological filters to identify meta-analyses and/or longitudinal/controlled studies;
• Narrowing search results by adding additional PICOC terms;
• Limit a search result by limiting the date range.

Chapter 6:

A short introduction to science

• Assess whether a study was conducted according to the scientific method;
• Recognize pseudo-science; Assess whether a statistically significant finding is of practical relevance;
• Assess whether methodological bias may have affected the results;
• Determine whether confounders may have affected the results;
• Assess whether a placebo effect may have affected the results;
• Identify moderators or mediators that may have affected the results;
• Distinguish quantitative research methods from qualitative research methods;
• Determine a study’s research design;
• Efficiently read a research paper.

Chapter 7:

APPRAISE – scientific evidence

• Assess the impact of an effect size;
• Assess whether a statistically significant finding is of practical relevance;
• Assess whether a confidence interval is sufficiently narrow;
• Assess whether an outcome was measured in a reliable way;
• Distinguish cause-and-effect questions from non-effect questions;
• Determine a study’s research design;
• Assessing whether a study’s research design is appropriate given the research question (methodological appropriateness);
• Assessing a study’s methodological quality;
• Grading a study’s trustworthiness on the basis of its methodological appropriateness and quality;
• Summarizing a study’s main findings, weaknesses, and overall trustworthiness.

Chapter 8:

ACQUIRE – organizational evidence

• Explain the added value of organizational evidence;
• Distinguish data, from information and evidence;
• Determine what organizational evidence to acquire;
• Determine what types of organizational evidence are available and where they are kept;
• Distinguish ‘normal’ data from Big Data;
• Determine where to find relevant organizational evidence;
• Determine whether the evidence concerns operational data, metrics, KPIs or benchmarks;
• Explain the difference between descriptive and inferential measures;
• Identify potential barriers to acquire organizational evidence.

Chapter 9:

APPRAISE – organizational evidence

• Determine whether a logic model was used to collect and analyze evidence from the organization;
• Assess whether organizational data are relevant;
• Identify steps in the collection and processing of data that could introduce risk of inaccurate data;
• Determine whether contextual information is missing;
• Determine whether there could be measurement error;
• Assess whether there could be a small number problem;
• Determine whether a metric is a good representation of the data;
• Interpret a metric’s standard deviation;
• Assess whether a graph represents the data in a valid and reliable way;
• Interpret a correlation or regression coefficient;
• Determine whether a correlation- or regression coefficient is practically relevant;
• Assess whether there are outliers that may distort the evidence;
• Assessing whether range restriction may have affected the evidence;
• Assess whether a confidence interval is sufficiently narrow.

Chapter 10:

ACQUIRE – stakeholder evidence

• Identify and assess evidence from stakeholders;
• Identify and distinguish different types of stakeholders;
• Determine which stakeholders could be affected by a decision;
• Determine which stakeholders could affect a decision, its implementation, or its outcome;
• Identify the most relevant stakeholders;
• Determine how to acquire evidence from stakeholders in a valid and reliable way.

Chapter 11:

APPRAISE – stakeholder evidence

• Explain why stakeholders’ subjective feelings and perceptions should always be taken into account;
• Determine the practical and/or ethical impact a decision may have on stakeholders;
• Determine whether relevant stakeholders can freely express their views and feelings regarding a (proposed) decision;
• Determine whether there could have been selection bias in the way evidence from stakeholders was obtained;
• Determine whether the evidence from stakeholders is sufficiently representative.

Chapter 12:

AGGREGATE

• Explain what proof, evidence, chance, and ‘conditional’ probability means;
• Assess the impact of a prior probability;
• Estimate the likelihood of the evidence: P(E|Htrue) and P(E|Hfalse);
• Update the probability of a claim/assumption/hypothesis when new evidence comes available;
• Aggregate evidence from multiple sources by applying Bayes Rule.

Chapter 13:

APPLY

• Use the PICOC method to determine whether the evidence applies to the organizational context;
• Determine whether a decision/intervention gives you the biggest bang for your buck;
• Assess the level of risk inherent in a decision/intervention;
• Identify ethical issues that need to be considered;
• Determine whether (and if so, how) the evidence is actionable;
• Determine whether there are moderators that need to be taken into account;
• Determine, given the type of decision at hand, how and in what form the evidence can be applied.

Chapter 14:

ASSESS

• Identify the type of decision (to be) made (routine, non-routine, or novel/hyper complex);
• Determine whether a decision was executed as planned;
• Assess an outcome using the gold standard method;
• Assessing an outcome using quasi- or non-experimental methods;
• Suggest ways to improve the validity and reliability of an outcome assessment;
• Assess whether an outcome was measured in a reliable way;
• Assess whether indirect and intangible costs were taken into account;
• Assess the (unintended) consequences of a decision on stakeholders.

Chapter 15:

Building EBMgt capacity in your organization

• Explain how to develop a questioning mindset;
• Explain how to make more mindful and explicit decisions more explicit;
• Create opportunities for people to practice and apply learning;
• Explain why evidence alone does not change people’s minds;
• Illustrate how to give people’s minds a way out;
• Help people to develop a new operating logic;
• Explain why increasing people’s accountability leads to more information-seeking behaviour;
• Explain why it is important to take small steps;
• Determine the best way to build evidence-based capacity in your organization.