Teaching and Learning blog

In recent years, researchers have increasingly advocated for open science (some scholars capitalize it as Open Science). The open science movement consists of several practices that make research reports more transparent, such that details regarding methodologies, collected data, and data analysis procedures are fully disclosed and open to public inspection and critique.

Here are five common ways in which researchers can enhance the transparency — and ultimately also the credibility — of their research projects. These points are described in greater depth in my book Practical Research: Design and Process, 13th Edition:

• Preregistration of a study: Researchers post an online description of what they will do in an upcoming research project. Among other things, such a description usually includes the overarching research question(s) to be addressed, specific hypotheses to be tested, specific methods to be used to collect data, and specific statistical procedures and other strategies to be used in analyzing the data. (See Chapter 5.)
• Results-blind reviewing: Before a project is conducted, outside reviewers examine a preregistered research proposal to determine the apparent soundness of a researcher’s main question(s), rationale, a priori hypotheses, study design, and planned data-collection and data-analysis strategies. (See Chapter 13.)
• Registered report: When a proposed project has been both (a) preregistered and (b) results-blind reviewed, an editor of an academic journal commits to publishing the final research report in that journal — regardless of whether the results are statistically significant or in some other way especially noteworthy — as long as the project has actually been carried out as originally proposed or else modified in reasonable, well-documented ways. (Again, see Chapter 13.)
• Open access to specific data collection and analysis procedures: Researchers make the details of their data-collection and data-analysis strategies readily available to interested scholars; for example, they might share the questionnaires administered, specific statistical procedures conducted, or particular coding schemes used to find patterns in participants’ interview responses. (Again, see Chapter 13.)
• Open access to raw data: Especially when qualitative data has been collected (e.g. when people have been interviewed or members of a particular sociocultural group have been carefully observed as they’ve gone about their daily activities), the actual verbal and/or nonverbal behaviors might be presented in an appendix or supplemental materials. There’s an important caveat here: A researcher can give other people open access to their raw data only when either (a) responses remain strictly anonymous or (b) participants have given explicit written permission for their identities to be disclosed. (Once again see Chapter 13; also, see the section “Ensuring Participants’ Rights and Well-Being” in Chapter 4.)

When striving for transparency, researchers don’t necessarily do all of these things for a particular research project. For example, when a researcher is building on another researcher’s work and uses a questionnaire that the other researcher has created, publishing the questionnaire might violate the other researcher’s intellectual property rights. And preregistration of a project isn’t always logistically feasible. For instance, many action research projects may be intentionally ill-defined at the very beginning, at least in part because initial data collection may lead to new questions to be addressed and further data collection may be necessary.

The bottom line here is this: I recommend open science practices when they can enhance the credibility of a research project without jeopardizing either (a) the privacy and well-being of the people being studied in the project or (b) the degree to which the project can thoroughly address the questions that underlie and drive the project.

In much of the 20th century, most researchers published results and conclusions with the hopes that other individuals would translate their findings into effective, “real-world” practices and interventions. In recent decades, however, many researchers have wanted applications of their findings to be integral parts of their own research projects. In particular, the term action research refers to projects that are designed not only (a) to make sense of an issue or problem but also (b) to take action and make concrete changes in conditions, practices, resources, and/or policies. In a subgroup of action research designs known as participatory designs, some researchers include one or more practitioners or other key stakeholders — people who can make use of a study’s findings in their future decision-making and intervention efforts — on their research teams.

Action research is almost invariably eclectic in its use of specific data collection strategies. For example, research projects might incorporate a combination of such strategies as surveys (face-to-face, paper-pencil, and/or online), individual and small-group interviews (e.g. focus groups), in-depth case studies, observations of people’s behaviors in real-life environments, and comparisons of groups in distinctly different educational or therapeutic settings.

Many action research projects are iterative in nature; that is, researchers move back and forth among various steps in the research process. For example, when researchers begin to analyze their data — or even before that, when they are still collecting data — they may find that the issue they have been addressing is more complex and multifaceted than they initially realized, and so they may go back and reformulate the questions they need to address, the data they need to gather, and the means by which they can gather that data.

Another important way in which action research tends to differ from more traditional research methodologies is in researchers’ dissemination strategies — that is, the ways in which researchers try to get the word out about their findings and potential implications. Traditional researchers typically describe their studies and findings in journal articles, books or book chapters, conference presentations, and — for graduate students — master’s theses and doctoral dissertations. Such reports can be quite effective in communicating findings to individuals working in the same or a similar field at colleges, universities, and other research settings, but the great majority of them escape the attention of practitioners, policy makers, and the public at large. Hence, action researchers often use additional, more local, strategies to broaden the audience that has access to their findings. Examples are websites, webinars, chatrooms, blogs, newsletters, in-person community forums and panel discussions, and popular social media platforms (e.g. Facebook, TikTok).

As we human beings learn new things every day, we all have ideas about what “knowledge” and “learning” are—ideas that are collectively known as epistemic beliefs. These beliefs typically include beliefs about many or all of the following:

• The certainty of knowledge: Whether knowledge is a fixed, unchanging, absolute “truth” or, instead, a tentative, dynamic entity that will continue to evolve over time.
• The simplicity and structure of knowledge: Whether knowledge is a collection of discrete, independent facts or, instead, a set of complex and interrelated ideas.
• The source of knowledge: Whether knowledge comes from outside of learners (i.e., from a teacher or other authority figure) or, instead, is derived and constructed by learners themselves.
• The criteria for determining truth: Whether an idea is accepted as true when it’s communicated by an expert or, instead, when it’s logically evaluated based on available evidence.
• The speed of learning: Whether knowledge is acquired quickly, if at all (in which case learners either know something or they don’t, in an all-or-none fashion) or, instead, is acquired gradually over a period of time (in which case learners can partially know something).
• The nature of learning ability: Whether people’s ability to learn is fixed at birth (i.e., inherited) or, instead, can improve over time with practice and use of better strategies.

Keep in mind that epistemic beliefs aren’t as either–or as I’ve just portrayed them. Most or all of the dimensions I’ve listed are probably continuums rather than strict either–or dichotomies.

Psychologists often use certain terms when referring to various beliefs about the nature of knowledge. Typical of 3-year-olds is a realist view, in which knowledge is the same as what people say or do (e.g., if I tell you that some cows have purple fur with orange spots, you’ll take my word for it). Four-year-olds are more likely to have an absolutist view, in which knowledge isn’t necessarily the same as people’s thoughts or assertions but it’s certain and definite—things are either absolutely right or absolutely wrong. Later on—typically in adolescence at the earliest—some individuals acquire a multiplist view, in which some knowledge is seen as uncertain, with people’s varying opinions all having equal legitimacy. People may or may not eventually acquire an evaluativist view, in which people’s ideas and opinions have more or less merit and legitimacy depending on whether defensible evidence or logic supports them.¹

It makes sense to hold an absolutist view about some kinds of knowledge. Certain bits of information are fairly black and white; we usually think of them as “facts.” For example, France is a country in Europe, Christopher Columbus first sailed across the Atlantic in 1492, and two things plus two more things give us four things altogether; these facts are unlikely to change in the foreseeable future. In other situations, a multiplist view makes sense. For example, there isn’t necessarily a single “right” answer to questions such as “What qualities are essential for ‘good’ music?” and “Is it appropriate to burp when you’re a dinner guest in someone else’s home?”

When conducting research on complex issues or problems, however, good researchers adapt an evaluatist perspective: They recognize that a particular premise or conclusion is probably “true” only to the extent that concrete evidence and logic support it. Accordingly, taking an evaluatist view requires researchers to engage in at least three mental processes:

• Critical thinking. Good researchers never take the things they read or hear at face value. Critical thinking involves evaluating the accuracy, credibility, and worth of information and lines of reasoning. For example, when people read about other individuals’ theories and research findings, they regularly ask themselves such questions as these: “Are there potential shortcomings in this research study that make me question the validity of the researcher’s conclusions?” “Does this researcher’s explanation make sense based on other research findings related to the issue being investigated?” “How might I improve on the research methods used in this study?”
• Metacognitive reflectiveness. The term metacognition means “thinking about the nature of thinking,” and metacognitive reflectiveness means “thinking about one’s own thinking.” Good researchers regularly reflect on their own thought processes, mentally checking themselves regarding their own logic. For example, they continually ask themselves whether they’re being as objective as possible in their observations, whether their evidence adequately supports their hypotheses and conclusions, and where there might be holes or inconsistencies in the theories they have constructed to explain a phenomenon they are investigating. Metacognitive reflectiveness, then, requires considerable critical thinking.
• Conceptual change when warranted. Conceptual change involves significantly revising one’s existing beliefs about a topic, enabling new, discrepant information to be better understood and explained. Good researchers regularly revise their beliefs, understandings, and explanations as credible new evidence and theories appear on the scene. In general, they keep open minds about the true nature of the phenomena they are investigating. Researchers who do otherwise—those who stubbornly stick to their own previous explanations even in the face of considerable contradictory information—impede scientific progress as we collectively strive to better understand our physical, psychological, and social worlds.

1 For groundbreaking work on this developmental trend, I refer you to two book chapters by Deanna Kuhn and colleagues:

• Kuhn, D., & Franklin, S. (2006). The second decade: What develops (and how)? In W. Damon & R. M. Lerner (Series Eds.), D. Kuhn & R. Siegler (Vol. Eds.), Handbook of child psychology: Vol. 2. Cognition, perception, and language (6th ed., pp. 953–993). New York, NY: Wiley.

• Kuhn, D., & Weinstock, M. (2002). What is epistemological thinking and why does it matter? In B. K. Hofer & P. R. Pintrich (Eds.), Personal epistemology: The psychology of beliefs about knowledge and knowing (pp. 121–144). Mahwah, NJ: Erlbaum

Even in graduate school, many students have misunderstandings about the nature of both science and scientific research. In this brief blog post, I hope to correct certain misconceptions my readers might have about these two concepts.

What does true scientific research entail?

In Chapter 1 of the book, Practical Research: Design and Process, 13th Edition, I identify activities that do not constitute true research as scientists use the term, and they’re worth repeating here. In particular, scientific research is not simply:

• Gathering information that already exists but is currently unknown to an individual making an inquiry about a topic.
• Rummaging around for existing information that is hard to find but is important for decision-making regarding an important project or problem.
• Transporting information from one location to another — for example, by writing a “research report” that a student submits to meet a course requirement.

Instead, true scientific research is a process of systematically collecting, analyzing, and interpreting data to enhance understanding of a particular phenomenon. Let’s look at specific words within this definition:

• Scientific research is systematic — that is, it is conducted in a somewhat preplanned, organized fashion, rather than being a totally rudderless, willy-nilly endeavor. As you will see, exactly how systematic a research endeavor is depends on the research methodology being used; for example, on average, quantitative research is more preplanned and systematic than qualitative research.
• Scientific research involves collecting new data — perhaps measurements of a certain physical phenomenon, people’s responses to a certain survey, or observations of commonplace behaviors within a certain social or cultural group. (Please note an exception here: Consistent with open science practices, a researcher might make use of another researcher’s data in a quest for additional insights that those data may yield.)
• Scientific research also involves analyzing a body of data to find patterns within it — patterns that might shed light on a phenomenon that has previously been poorly understood.
• Finally, scientific research involves interpreting and trying to make sense of those data — going beyond the data themselves to draw conclusions about the underlying phenomenon being studied.

Driving the whole research endeavor are one or more research problems or questions that the researcher is trying to address and potentially solve. In Chapter 1 I describe such problems and questions, and in Chapter 2 I give you considerable guidance about how to identify appropriate ones for your own research projects.

What, exactly, is science?

When children first hear the term “science” in elementary and secondary school grades, their teachers are typically referring to biology, chemistry, physics, astronomy, and the like. But in fact, most academic disciplines — not only the biological and physical sciences, but also the psychological and social sciences — have elements of science at their core.

The helping professions, too, are rooted in science. Medicine and health care are rooted in biology, but so are education, psychotherapy, social work, coaching, and many other helping professions dependent on scientific findings.

Here I simply want to highlight three things that a scientific approach to a problem or question typically involves, plus one thing that it definitely does not involve:

• Hypothesis formation and testing: When conducting a research project, many scientists begin with one or more tentative hypotheses regarding processes or dynamics that underlie the phenomenon under investigation; they will then design a study that can enable them to systematically test those hypotheses. In some cases, however, researchers might decide not to form any hypotheses at the outset, typically in an effort to be as open-minded as possible about what the collected data might reveal. Nevertheless, in virtually all scientific research, one or more hypotheses eventually emerge as potential explanations for the phenomenon at hand, and at some point during a project, those hypotheses are tested.
• Construction of theories and/or models: Scientific inquiry is a very constructive process. Often it involves constructing or revising a theory—an integrated set of concepts and principles developed to explain a particular phenomenon. Alternatively, it might involve constructing a model—a physical or graphical representation that shows how certain entities might be interrelated parts of a larger system. As examples, you’ve undoubtedly seen physical models of the sun and various planets in our solar system, and you’ve seen graphic models of various phenomena (perhaps including cause-and-effect relationships) in college textbooks.
• Some degree of objectivity: Researchers strive to be as objective as they can in their quest to collect, analyze, and interpret data. Unfortunately, thanks to the various expectations, biases, mindsets, and worldviews that researchers inevitably bring to any research project, complete objectivity is virtually impossible. As you read the book, you’ll discover the many ways in which the human mind can distort the true “realities” of our physical, psychological, and social worlds — if such realities exist at all. You’ll also discover that, especially in the psychological and social sciences, many researchers have acknowledged that they can never be completely objective in their research findings and conclusions.
• Recognition that it is virtually impossible to prove anything beyond a shadow of a doubt: As our international scientific community increasingly adds new information to our knowledge base about various physical, psychological, and social phenomena, we can continue to zero in on things that might be true about the world and the larger universe in which we live. But for logistical and statistical reasons, we will probably never know what things are actually true. For example, as a high school student back in the 1960s, I was taught that atoms and the protons, neutrons, and electrons within them were the smallest components of the physical universe. But then along came quarks, muons, neutrinos, and other entities that are seemingly even smaller.

I should mention here that the impossibility of proving something doesn’t mean that the theories and models that scientists construct to explain their findings have no merit. As more and more data related to a particular phenomenon is collected, some theories and models are regularly revised to take that data into account; meanwhile, other theories and models might be discarded because they cannot adequately account for the new data.

Even though a given theory cannot ultimately be proven to be a complete explanation of a phenomenon, it can nevertheless have considerable usefulness in making predictions and identifying interventions that can enhance the well-being of human beings, other living things, and the planet on which we all live.

Researchers’ particular beliefs about the nature of human knowledge can have a major impact on the quality of the research they conduct, the conclusions they draw, and, ultimately, the impact they have on our communal knowledge of the world as a whole. I expand on this point in a separate blog post: “How researchers’ epistemic beliefs influence the quality of their work.”

Researchers’ beliefs about legitimate and credible research designs, methodologies, processes, and practices have expanded in a great many ways since the 1970s, and I — who obtained my bachelor’s, master’s, and doctoral degrees in the 1970s — have observed the continual expansion over the past several decades with both awe and delight. There is so much more that we, as a larger society, can learn when we look at puzzling phenomena from multiple angles and perspectives!

Noteworthy Changes in Practical Research, 13th Edition (2024)

To capture recent trends in research methodologies and perspectives, I have made several noteworthy changes to the content of the latest edition:

• I have significantly reorganized and greatly expanded discussions of various action research and participatory designs (see Chapter 10). A separate blog post presents a brief description of action research.
• I have increased discussions of the ethics of research and possible biases that might adversely affect the quality of a research project and/or research report (see Chapters 4, 6, 7, 8, 9, 10, 11, 12, and 13).
• I have added discussions of open science practices (see Chapters 5 and 13). A separate blog post provides a short overview of what open science can entail.
• I have included several new illustrative examples of research methodologies (see Chapters 8, 9, and 10).
• With the assumption that my readers are now more technologically literate than they were even a few years ago, I have updated discussions of technology-based strategies (e.g. new software options) while cutting back on coverage of more elementary strategies (e.g. how to use word-processing software).
• I have added five new Conceptual Analysis features with which readers can self-assess their understanding of key concepts and principles in the book (see Chapters 1, 6, 11, and 12).
• At the end of each chapter, I have added a short summary that can help readers mentally revisit and review central ideas within the chapter.

Beginning with the 12th edition, the book no longer includes a chapter on historical research. (With significant advances in other research methodologies, there was simply no longer any room in the book for it.) Yet historical research methodologies are essential for students in history and related disciplines and can also be useful in other disciplines. Readers who would like to learn more about historical research can find an entire chapter on the topic in the book’s 11th edition, published in 2016.

Two changes in the latest, 13th edition of Practical Research appear on the book’s cover and title page. First, the subtitle of the book, previously Planning and Design, has changed to Design and Process. Although this change might strike my readers as a subtle one, it reflects the ways in which the book’s contents have evolved over the years to focus increasingly on the many, many simultaneous and sequential processes — not only physical processes but mental processes as well — that underlie well-designed, high-quality research endeavors.

Second, whereas Paul Leedy and I have been coauthors of the past several editions of the book, I am now listed as sole author. To explain this change, I must go back to the very first edition. In the early 1970s, as a professor at American University, Paul saw the need for a research methods book that provided specific, concrete guidance for students and other novice researchers who were working on independent research projects (e.g. projects for masters’ theses and doctoral dissertations). The result was the first edition of the book, published in 1974.

As various research methodologies continued to evolve in many academic disciplines, Paul updated the book in 2nd, 3rd, 4th, 5th, and 6th editions (with contributions by Tim Newby, Peggy Ertmer, and the editorial staff in the 6th edition).

With Paul’s eventual retirement, his editor, Kevin Davis, asked me to take over, beginning with the 7th edition. Paul’s ideas and words have always been key elements of the book, however, and thus it was quite appropriate to list us as coauthors. But as I wrote the 13th edition, the book had changed so much that it made sense to Pearson’s editorial staff that I become sole author.