problem solving ability test introduction

What Is Problem Solving? How Software Engineers Approach Complex Challenges

From debugging an existing system to designing an entirely new software application, a day in the life of a software engineer is filled with various challenges and complexities. The one skill that glues these disparate tasks together and makes them manageable? Problem solving .

Throughout this blog post, we’ll explore why problem-solving skills are so critical for software engineers, delve into the techniques they use to address complex challenges, and discuss how hiring managers can identify these skills during the hiring process.

What Is Problem Solving?

But what exactly is problem solving in the context of software engineering? How does it work, and why is it so important?

Problem solving, in the simplest terms, is the process of identifying a problem, analyzing it, and finding the most effective solution to overcome it. For software engineers, this process is deeply embedded in their daily workflow. It could be something as simple as figuring out why a piece of code isn’t working as expected, or something as complex as designing the architecture for a new software system.

In a world where technology is evolving at a blistering pace, the complexity and volume of problems that software engineers face are also growing. As such, the ability to tackle these issues head-on and find innovative solutions is not only a handy skill — it’s a necessity.

The Importance of Problem-Solving Skills for Software Engineers

Problem-solving isn’t just another ability that software engineers pull out of their toolkits when they encounter a bug or a system failure. It’s a constant, ongoing process that’s intrinsic to every aspect of their work. Let’s break down why this skill is so critical.

Driving Development Forward

Without problem solving, software development would hit a standstill. Every new feature, every optimization, and every bug fix is a problem that needs solving. Whether it’s a performance issue that needs diagnosing or a user interface that needs improving, the capacity to tackle and solve these problems is what keeps the wheels of development turning.

It’s estimated that 60% of software development lifecycle costs are related to maintenance tasks, including debugging and problem solving. This highlights how pivotal this skill is to the everyday functioning and advancement of software systems.

Innovation and Optimization

The importance of problem solving isn’t confined to reactive scenarios; it also plays a major role in proactive, innovative initiatives . Software engineers often need to think outside the box to come up with creative solutions, whether it’s optimizing an algorithm to run faster or designing a new feature to meet customer needs. These are all forms of problem solving.

Consider the development of the modern smartphone. It wasn’t born out of a pre-existing issue but was a solution to a problem people didn’t realize they had — a device that combined communication, entertainment, and productivity into one handheld tool.

Increasing Efficiency and Productivity

Good problem-solving skills can save a lot of time and resources. Effective problem-solvers are adept at dissecting an issue to understand its root cause, thus reducing the time spent on trial and error. This efficiency means projects move faster, releases happen sooner, and businesses stay ahead of their competition.

Improving Software Quality

Problem solving also plays a significant role in enhancing the quality of the end product. By tackling the root causes of bugs and system failures, software engineers can deliver reliable, high-performing software. This is critical because, according to the Consortium for Information and Software Quality, poor quality software in the U.S. in 2022 cost at least $2.41 trillion in operational issues, wasted developer time, and other related problems.

Problem-Solving Techniques in Software Engineering

So how do software engineers go about tackling these complex challenges? Let’s explore some of the key problem-solving techniques, theories, and processes they commonly use.

Decomposition

Breaking down a problem into smaller, manageable parts is one of the first steps in the problem-solving process. It’s like dealing with a complicated puzzle. You don’t try to solve it all at once. Instead, you separate the pieces, group them based on similarities, and then start working on the smaller sets. This method allows software engineers to handle complex issues without being overwhelmed and makes it easier to identify where things might be going wrong.

Abstraction

In the realm of software engineering, abstraction means focusing on the necessary information only and ignoring irrelevant details. It is a way of simplifying complex systems to make them easier to understand and manage. For instance, a software engineer might ignore the details of how a database works to focus on the information it holds and how to retrieve or modify that information.

Algorithmic Thinking

At its core, software engineering is about creating algorithms — step-by-step procedures to solve a problem or accomplish a goal. Algorithmic thinking involves conceiving and expressing these procedures clearly and accurately and viewing every problem through an algorithmic lens. A well-designed algorithm not only solves the problem at hand but also does so efficiently, saving computational resources.

Parallel Thinking

Parallel thinking is a structured process where team members think in the same direction at the same time, allowing for more organized discussion and collaboration. It’s an approach popularized by Edward de Bono with the “ Six Thinking Hats ” technique, where each “hat” represents a different style of thinking.

In the context of software engineering, parallel thinking can be highly effective for problem solving. For instance, when dealing with a complex issue, the team can use the “White Hat” to focus solely on the data and facts about the problem, then the “Black Hat” to consider potential problems with a proposed solution, and so on. This structured approach can lead to more comprehensive analysis and more effective solutions, and it ensures that everyone’s perspectives are considered.

This is the process of identifying and fixing errors in code . Debugging involves carefully reviewing the code, reproducing and analyzing the error, and then making necessary modifications to rectify the problem. It’s a key part of maintaining and improving software quality.

Testing and Validation

Testing is an essential part of problem solving in software engineering. Engineers use a variety of tests to verify that their code works as expected and to uncover any potential issues. These range from unit tests that check individual components of the code to integration tests that ensure the pieces work well together. Validation, on the other hand, ensures that the solution not only works but also fulfills the intended requirements and objectives.

Explore verified tech roles & skills.

The definitive directory of tech roles, backed by machine learning and skills intelligence.

Explore all roles

Evaluating Problem-Solving Skills

We’ve examined the importance of problem-solving in the work of a software engineer and explored various techniques software engineers employ to approach complex challenges. Now, let’s delve into how hiring teams can identify and evaluate problem-solving skills during the hiring process.

Recognizing Problem-Solving Skills in Candidates

How can you tell if a candidate is a good problem solver? Look for these indicators:

Previous Experience: A history of dealing with complex, challenging projects is often a good sign. Ask the candidate to discuss a difficult problem they faced in a previous role and how they solved it.
Problem-Solving Questions: During interviews, pose hypothetical scenarios or present real problems your company has faced. Ask candidates to explain how they would tackle these issues. You’re not just looking for a correct solution but the thought process that led them there.
Technical Tests: Coding challenges and other technical tests can provide insight into a candidate’s problem-solving abilities. Consider leveraging a platform for assessing these skills in a realistic, job-related context.

Assessing Problem-Solving Skills

Once you’ve identified potential problem solvers, here are a few ways you can assess their skills:

Solution Effectiveness: Did the candidate solve the problem? How efficient and effective is their solution?
Approach and Process: Go beyond whether or not they solved the problem and examine how they arrived at their solution. Did they break the problem down into manageable parts? Did they consider different perspectives and possibilities?
Communication: A good problem solver can explain their thought process clearly. Can the candidate effectively communicate how they arrived at their solution and why they chose it?
Adaptability: Problem-solving often involves a degree of trial and error. How does the candidate handle roadblocks? Do they adapt their approach based on new information or feedback?

Hiring managers play a crucial role in identifying and fostering problem-solving skills within their teams. By focusing on these abilities during the hiring process, companies can build teams that are more capable, innovative, and resilient.

Key Takeaways

As you can see, problem solving plays a pivotal role in software engineering. Far from being an occasional requirement, it is the lifeblood that drives development forward, catalyzes innovation, and delivers of quality software.

By leveraging problem-solving techniques, software engineers employ a powerful suite of strategies to overcome complex challenges. But mastering these techniques isn’t simple feat. It requires a learning mindset, regular practice, collaboration, reflective thinking, resilience, and a commitment to staying updated with industry trends.

For hiring managers and team leads, recognizing these skills and fostering a culture that values and nurtures problem solving is key. It’s this emphasis on problem solving that can differentiate an average team from a high-performing one and an ordinary product from an industry-leading one.

At the end of the day, software engineering is fundamentally about solving problems — problems that matter to businesses, to users, and to the wider society. And it’s the proficient problem solvers who stand at the forefront of this dynamic field, turning challenges into opportunities, and ideas into reality.

This article was written with the help of AI. Can you tell which parts?

Get started with HackerRank

Over 2,500 companies and 40% of developers worldwide use HackerRank to hire tech talent and sharpen their skills.

What Factors Actually Impact a Developer’s Decision to Accept an Offer?

Find out how Siemens has benefited from our services

of the DAX 30 companies work with us

Solutions Use cases Answer to your HR questions. smartData Market Intelligence Access to the world’s largest labor market database to tune your business and HR. smartPlan Future Workforce Planning Design your future workforce & uncover skills risks and gaps. smartPeople Skills Fulfillment Discover your internal skills and build a future-fit workforce.
Podcasts & interviews
ROI calculator
HR Glossary

See why 100+ companies choose HRForecast.

Book a demo

How to assess problem-solving skills

Human beings have been fascinated and motivated by problem-solving for as long as time. Let’s start with the classic ancient legend of Oedipus. The Sphinx aggressively addressed anyone who dared to enter Thebes by posing a riddle. If the traveler failed to answer the riddle correctly, the result was death. However, the Sphinx would be destroyed when the answer was finally correct.

Alas, along came Oedipus. He answered correctly. He unlocked this complex riddle and killed the Sphinx.

However, rationality was hardly defined at that time. Today, though, most people assume that it simply takes raw intelligence to be a great problem solver. However, it’s not the only crucial element.

Introduction to key problem-solving skills

You’ve surely noticed that many of the skills listed in the problem-solving process are repeated. This is because having these abilities and talents are so crucial to the entire course of getting a problem solved. Let’s look at some key problem-solving skills that are essential in the workplace.

Communication, listening, and customer service skills

In all the stages of problem-solving, you need to listen and engage to understand what the problem is and come to a conclusion as to what the solution may be. Another challenge is being able to communicate effectively so that people understand what you’re saying. It further rolls into interpersonal communication and customer service skills, which really are all about listening and responding appropriately.

Data analysis, research, and topic understanding skills

To produce the best solutions, employees must be able to understand the problem thoroughly. This is possible when the workforce studies the topic and the process correctly. In the workplace, this knowledge comes from years of relevant experience.

Dependability, believability, trustworthiness, and follow-through

To make change happen and take the following steps towards problem-solving, the qualities of dependability, trustworthiness, and diligence are a must. For example, if a person is known for not keeping their word, laziness, and committing blunders, that is not someone you’ll depend on when they provide you with a solution, will you?

Leadership, team-building, and decision-making

A true leader can learn and grow from the problems that arise in their jobs and utilize each challenge to hone their leadership skills. Problem-solving is an important skill for leaders who want to eliminate challenges that can otherwise hinder their people’s or their business’ growth. Let’s take a look at some statistics that prove just how important these skills are:

A Harvard Business Review study states that of all the skills that influence a leader’s success, problem-solving ranked third out of 16.

According to a survey by Goremotely.net, only 10% of CEOs are leaders who guide staff by example .

Another study at Havard Business Review found a direct link between teambuilding as a social activity and employee motivation.

Are you looking for a holistic way to develop leaders in your workplace?

Numerous skills and attributes define a successful one from a rookie when it comes to leaders. Our leadership development plan (with examples!) can help HR leaders identify potential leaders that are in sync with your company’s future goals.

Why is problem solving important in the workplace?

As a business leader, when too much of your time is spent managing escalations, the lack of problem-solving skills may hurt your business. While you may be hiring talented and capable employees and paying them well, it is only when you harness their full potential and translate that into business value that it is considered a successful hire.

The impact of continuing with poor problem-solving skills may show up in your organization as operational inefficiencies that may also manifest in product quality issues, defects, re-work and non-conformance to design specifications. When the product is defective, or the service is not up to the mark, it directly affects your customer’s experience and consequently reflects on the company’s profile.

At times, poor problem-solving skills could lead to missed market opportunities, slow time to market, customer dissatisfaction, regulatory compliance issues, and declining employee morale.

Problem-solving skills are important for individual business leaders as well. Suppose you’re busy responding to frequent incidents that have the same variables. In that case, this prevents you from focusing your time and effort on improving the future success of business outcomes.

Proven methods to assess and improve problem-solving skills

Pre-employment problem-solving skill assessment .

Recent research indicates that up to 85% of resumes contain misleading statements. Similarly, interviews are subjective and ultimately serve as poor predictors of job performance .

To provide a reliable and objective means of gathering job-related information on candidates, you must validate and develop pre-employment problem-solving assessments. You can further use the data from pre-employment tests to make informed and defensible hiring decisions.

Depending on the job profile, below are examples of pre-employment problem-solving assessment tests:

Personality tests: The rise of personality testing in the 20th century was an endeavor to maximize employee potential. Personality tests help to identify workplace patterns, relevant characteristics, and traits, and to assess how people may respond to different situations.

Examples of personality tests include the Big five personality traits test and Mercer | Mettl’s Dark Personality Inventory .

Cognitive ability test: A pre-employment aptitude test assesses individuals’ abilities such as critical thinking, verbal reasoning, numerical ability, problem-solving, decision-making, etc., which are indicators of a person’s intelligence quotient (IQ). The test results provide data about on-the-job performance. It also assesses current and potential employees for different job levels.

Criteria Cognitive Aptitude test , McQuaig Mental Agility Test , and Hogan Business Reasoning Inventory are commonly used cognitive ability assessment tests.

Convergent and divergent thinking methods

American psychologist JP Guilford coined the terms “convergent thinking” and “divergent thinking” in the 1950s.

Convergent thinking involves starting with pieces of information and then converging around a solution. An example would be determining the correct answer to a multiple-choice question.

The nature of the question does not demand creativity but rather inherently encourages a person to consider the veracity of each answer provided before selecting the single correct one.

Divergent thinking, on the other hand, starts with a prompt that encourages people to think critically, diverging towards distinct answers. An example of divergent thinking would be asking open-ended questions.

Here’s an example of what convergent thinking and a divergent problem-solving model would look like.

The 5 whys method , developed by Sakichi Toyoda, is part of the Toyota production system. In this method, when you come across a problem, you analyze the root cause by asking “Why?” five times. By recognizing the countermeasure, you can prevent the problem from recurring. Here’s an example of the 5 whys method.

Source: Kanbanzie

This method is specifically useful when you have a recurring problem that reoccurs despite repeated actions to address it. It indicates that you are treating the symptoms of the problem and not the actual problem itself.

Starbursting

While brainstorming is about the team coming together to try to find answers, starbursting flips it over and asks everyone to think of questions instead. Here’s an example of the starbursting method.

The idea of this method is to go and expand from here, layering more and more questions until you’ve covered every eventuality of the problem.

Use of data analysis to measure improvement in problem-solving skills for your organization

Problem-solving and data analytics are often used together. Supporting data is very handy whenever a particular problem occurs. By using data analytics, you can find the supporting data and analyze it to use for solving a specific problem.

However, we must emphasize that the data you’re using to solve the problem is accurate and complete. Otherwise, misleading data may take you off track of the problem at hand or even make it appear more complex than it is. Moreover, as you gain knowledge about the current problem, it further eases the way to solve it.

Let’s dig deeper into the top 3 reasons data analytics is important in problem-solving.

1. Uncover hidden details

Modern data analytics tools have numerous features that let you analyze the given data thoroughly and find hidden or repeating trends without needing any extra human effort. These automated tools are great at extracting the depths of data, going back way into the past.

2. Automated models

Automation is the future. Businesses don’t have enough time or the budget to encourage manual workforces to go through loads of data to solve business problems. Instead, the tools can collect, combine, clean, and transform the relevant data all by themselves and finally use it to predict the solutions.

3. Explore similar problems

When you use a data analytics approach to solve problems, you can collect all the data available and store it. It can assist you when you find yourself in similar problems, providing references for how such issues were tackled in the past.

If you’re looking for ways to help develop problem-solving skills in the workplace and want to build a team of employees who can solve their own problems, contact us to learn how we can help you achieve it.

Stay up to date with our newsletter

Every month, we’ll send you a curated newsletter with our updates and the latest industry news.

How to measure your succession planning strategy: 4 metrics you need to track

pros and cons of internal vs external hiring

External vs internal hiring: pros and cons of every method

Why are design thinking and Agile methods so critical to your HR strategy?

The importance of design thinking and Agile methods in today’s HR

HRForecast newsletter

Get only relevant and insightful letters from us every month

Not a customer yet? Contact us

Career at HRForecast

Why hrforecast.

Customer Stories
Trust and Security
Data Analytics Approach
IT Skills Analytics
smartPeople

2023 © Copyright - HRForecast | Imprint | Privacy policy | Terms and conditions (MSA)

Introduction to Problem Solving Skills

What is problem solving and why is it important.

The ability to solve problems is a basic life skill and is essential to our day-to-day lives, at home, at school, and at work. We solve problems every day without really thinking about how we solve them. For example: it’s raining and you need to go to the store. What do you do? There are lots of possible solutions. Take your umbrella and walk. If you don't want to get wet, you can drive, or take the bus. You might decide to call a friend for a ride, or you might decide to go to the store another day. There is no right way to solve this problem and different people will solve it differently.

Problem solving is the process of identifying a problem, developing possible solution paths, and taking the appropriate course of action.

Why is problem solving important? Good problem solving skills empower you not only in your personal life but are critical in your professional life. In the current fast-changing global economy, employers often identify everyday problem solving as crucial to the success of their organizations. For employees, problem solving can be used to develop practical and creative solutions, and to show independence and initiative to employers.

Throughout this case study you will be asked to jot down your thoughts in idea logs. These idea logs are used for reflection on concepts and for answering short questions. When you click on the "Next" button, your responses will be saved for that page. If you happen to close the webpage, you will lose your work on the page you were on, but previous pages will be saved. At the end of the case study, click on the "Finish and Export to PDF" button to acknowledge completion of the case study and receive a PDF document of your idea logs.

What Does Problem Solving Look Like?

IDEAL heuristic strategy for problem solving

The ability to solve problems is a skill, and just like any other skill, the more you practice, the better you get. So how exactly do you practice problem solving? Learning about different problem solving strategies and when to use them will give you a good start. Problem solving is a process. Most strategies provide steps that help you identify the problem and choose the best solution. There are two basic types of strategies: algorithmic and heuristic.

Algorithmic strategies are traditional step-by-step guides to solving problems. They are great for solving math problems (in algebra: multiply and divide, then add or subtract) or for helping us remember the correct order of things (a mnemonic such as “Spring Forward, Fall Back” to remember which way the clock changes for daylight saving time, or “Righty Tighty, Lefty Loosey” to remember what direction to turn bolts and screws). Algorithms are best when there is a single path to the correct solution.

But what do you do when there is no single solution for your problem? Heuristic methods are general guides used to identify possible solutions. A popular one that is easy to remember is IDEAL [ Bransford & Stein, 1993 ] :

I dentify the problem
D efine the context of the problem
E xplore possible strategies
A ct on best solution

IDEAL is just one problem solving strategy. Building a toolbox of problem solving strategies will improve your problem solving skills. With practice, you will be able to recognize and use multiple strategies to solve complex problems.

Watch the video

What is the best way to get a peanut out of a tube that cannot be moved? Watch a chimpanzee solve this problem in the video below [ Geert Stienissen, 2010 ].

[PDF transcript]

Describe the series of steps you think the chimpanzee used to solve this problem.

[Page 2: What does Problem Solving Look Like?] Describe the series of steps you think the chimpanzee used to solve this problem.

Think of an everyday problem you've encountered recently and describe your steps for solving it.

[Page 2: What does Problem Solving Look Like?] Think of an everyday problem you've encountered recently and describe your steps for solving it.

Developing Problem Solving Processes

Problem solving is a process that uses steps to solve problems. But what does that really mean? Let's break it down and start building our toolbox of problem solving strategies.

What is the first step of solving any problem? The first step is to recognize that there is a problem and identify the right cause of the problem. This may sound obvious, but similar problems can arise from different events, and the real issue may not always be apparent. To really solve the problem, it's important to find out what started it all. This is called identifying the root cause .

Example: You and your classmates have been working long hours on a project in the school's workshop. The next afternoon, you try to use your student ID card to access the workshop, but discover that your magnetic strip has been demagnetized. Since the card was a couple of years old, you chalk it up to wear and tear and get a new ID card. Later that same week you learn that several of your classmates had the same problem! After a little investigation, you discover that a strong magnet was stored underneath a workbench in the workshop. The magnet was the root cause of the demagnetized student ID cards.

The best way to identify the root cause of the problem is to ask questions and gather information. If you have a vague problem, investigating facts is more productive than guessing a solution. Ask yourself questions about the problem. What do you know about the problem? What do you not know? When was the last time it worked correctly? What has changed since then? Can you diagram the process into separate steps? Where in the process is the problem occurring? Be curious, ask questions, gather facts, and make logical deductions rather than assumptions.

Watch Adam Savage from Mythbusters, describe his problem solving process [ ForaTv, 2010 ]. As you watch this section of the video, try to identify the questions he asks and the different strategies he uses.

Adam Savage shared many of his problem solving processes. List the ones you think are the five most important. Your list may be different from other people in your class—that's ok!

[Page 3: Developing Problem Solving Processes] Adam Savage shared many of his problem solving processes. List the ones you think are the five most important.

“The ability to ask the right question is more than half the battle of finding the answer.” — Thomas J. Watson , founder of IBM

Voices From the Field: Solving Problems

In manufacturing facilities and machine shops, everyone on the floor is expected to know how to identify problems and find solutions. Today's employers look for the following skills in new employees: to analyze a problem logically, formulate a solution, and effectively communicate with others.

In this video, industry professionals share their own problem solving processes, the problem solving expectations of their employees, and an example of how a problem was solved.

Meet the Partners:

Taconic High School in Pittsfield, Massachusetts, is a comprehensive, fully accredited high school with special programs in Health Technology, Manufacturing Technology, and Work-Based Learning.
Berkshire Community College in Pittsfield, Massachusetts, prepares its students with applied manufacturing technical skills, providing hands-on experience at industrial laboratories and manufacturing facilities, and instructing them in current technologies.
H.C. Starck in Newton, Massachusetts, specializes in processing and manufacturing technology metals, such as tungsten, niobium, and tantalum. In almost 100 years of experience, they hold over 900 patents, and continue to innovate and develop new products.
Nypro Healthcare in Devens, Massachusetts, specializes in precision injection-molded healthcare products. They are committed to good manufacturing processes including lean manufacturing and process validation.

Making Decisions

Now that you have a couple problem solving strategies in your toolbox, let's practice. In this exercise, you are given a scenario and you will be asked to decide what steps you would take to identify and solve the problem.

Scenario: You are a new employee and have just finished your training. As your first project, you have been assigned the milling of several additional components for a regular customer. Together, you and your trainer, Bill, set up for the first run. Checking your paperwork, you gather the tools and materials on the list. As you are mounting the materials on the table, you notice that you didn't grab everything and hurriedly grab a few more items from one of the bins. Once the material is secured on the CNC table, you load tools into the tool carousel in the order listed on the tool list and set the fixture offsets.

Bill tells you that since this is a rerun of a job several weeks ago, the CAD/CAM model has already been converted to CNC G-code. Bill helps you download the code to the CNC machine. He gives you the go-ahead and leaves to check on another employee. You decide to start your first run.

What problems did you observe in the video?

[Page 5: Making Decisions] What problems did you observe in the video?
What do you do next?
Try to fix it yourself.
Ask your trainer for help.

As you are cleaning up, you think about what happened and wonder why it happened. You try to create a mental picture of what happened. You are not exactly sure what the end mill hit, but it looked like it might have hit the dowel pin. You wonder if you grabbed the correct dowel pins from the bins earlier.

You can think of two possible next steps. You can recheck the dowel pin length to make sure it is the correct length, or do a dry run using the CNC single step or single block function with the spindle empty to determine what actually happened.

Check the dowel pins.
Use the single step/single block function to determine what happened.

You notice that your trainer, Bill, is still on the floor and decide to ask him for help. You describe the problem to him. Bill asks if you know what the end mill ran into. You explain that you are not sure but you think it was the dowel pin. Bill reminds you that it is important to understand what happened so you can fix the correct problem. He suggests that you start all over again and begin with a dry run using the single step/single block function, with the spindle empty, to determine what it hit. Or, since it happened at the end, he mentions that you can also check the G-code to make sure the Z-axis is raised before returning to the home position.

Run the single step/single block function.
Edit the G-code to raise the Z-axis.

You finish cleaning up and check the CNC for any damage. Luckily, everything looks good. You check your paperwork and gather the components and materials again. You look at the dowel pins you used earlier, and discover that they are not the right length. As you go to grab the correct dowel pins, you have to search though several bins. For the first time, you are aware of the mess - it looks like the dowel pins and other items have not been put into the correctly labeled bins. You spend 30 minutes straightening up the bins and looking for the correct dowel pins.

Finally finding them, you finish setting up. You load tools into the tool carousel in the order listed on the tool list and set the fixture offsets. Just to make sure, you use the CNC single step/single block function, to do a dry run of the part. Everything looks good! You are ready to create your first part. The first component is done, and, as you admire your success, you notice that the part feels hotter than it should.

You wonder why? You go over the steps of the process to mentally figure out what could be causing the residual heat. You wonder if there is a problem with the CNC's coolant system or if the problem is in the G-code.

Look at the G-code.

After thinking about the problem, you decide that maybe there's something wrong with the setup. First, you clean up the damaged materials and remove the broken tool. You check the CNC machine carefully for any damage. Luckily, everything looks good. It is time to start over again from the beginning.

You again check your paperwork and gather the tools and materials on the setup sheet. After securing the new materials, you use the CNC single step/single block function with the spindle empty, to do a dry run of the part. You watch carefully to see if you can figure out what happened. It looks to you like the spindle barely misses hitting the dowel pin. You determine that the end mill was broken when it hit the dowel pin while returning to the start position.

After conducting a dry run using the single step/single block function, you determine that the end mill was damaged when it hit the dowel pin on its return to the home position. You discuss your options with Bill. Together, you decide the best thing to do would be to edit the G-code and raise the Z-axis before returning to home. You open the CNC control program and edit the G-code. Just to make sure, you use the CNC single step/single block function, to do another dry run of the part. You are ready to create your first part. It works. You first part is completed. Only four more to go.

As you are cleaning up, you notice that the components are hotter than you expect and the end mill looks more worn than it should be. It dawns on you that while you were milling the component, the coolant didn't turn on. You wonder if it is a software problem in the G-code or hardware problem with the CNC machine.

It's the end of the day and you decide to finish the rest of the components in the morning.

You decide to look at the G-code in the morning.
You leave a note on the machine, just in case.

You decide that the best thing to do would be to edit the G-code and raise the Z-axis of the spindle before it returns to home. You open the CNC control program and edit the G-code.

While editing the G-code to raise the Z-axis, you notice that the coolant is turned off at the beginning of the code and at the end of the code. The coolant command error caught your attention because your coworker, Mark, mentioned having a similar issue during lunch. You change the coolant command to turn the mist on.

You decide to talk with your supervisor.
You discuss what happened with a coworker over lunch.

As you reflect on the residual heat problem, you think about the machining process and the factors that could have caused the issue. You try to think of anything and everything that could be causing the issue. Are you using the correct tool for the specified material? Are you using the specified material? Is it running at the correct speed? Is there enough coolant? Are there chips getting in the way?

Wait, was the coolant turned on? As you replay what happened in your mind, you wonder why the coolant wasn't turned on. You decide to look at the G-code to find out what is going on.

From the milling machine computer, you open the CNC G-code. You notice that there are no coolant commands. You add them in and on the next run, the coolant mist turns on and the residual heat issues is gone. Now, its on to creating the rest of the parts.

Have you ever used brainstorming to solve a problem? Chances are, you've probably have, even if you didn't realize it.

You notice that your trainer, Bill, is on the floor and decide to ask him for help. You describe the problem with the end mill breaking, and how you discovered that items are not being returned to the correctly labeled bins. You think this caused you to grab the incorrect length dowel pins on your first run. You have sorted the bins and hope that the mess problem is fixed. You then go on to tell Bill about the residual heat issue with the completed part.

Together, you go to the milling machine. Bill shows you how to check the oil and coolant levels. Everything looks good at the machine level. Next, on the CNC computer, you open the CNC G-code. While looking at the code, Bill points out that there are no coolant commands. Bill adds them in and when you rerun the program, it works.

Bill is glad you mentioned the problem to him. You are the third worker to mention G-code issues over the last week. You noticed the coolant problems in your G-code, John noticed a Z-axis issue in his G-code, and Sam had issues with both the Z-axis and the coolant. Chances are, there is a bigger problem and Bill will need to investigate the root cause .

Talking with Bill, you discuss the best way to fix the problem. Bill suggests editing the G-code to raise the Z-axis of the spindle before it returns to its home position. You open the CNC control program and edit the G-code. Following the setup sheet, you re-setup the job and use the CNC single step/single block function, to do another dry run of the part. Everything looks good, so you run the job again and create the first part. It works. Since you need four of each component, you move on to creating the rest of them before cleaning up and leaving for the day.

It's a new day and you have new components to create. As you are setting up, you go in search of some short dowel pins. You discover that the bins are a mess and components have not been put away in the correctly labeled bins. You wonder if this was the cause of yesterday's problem. As you reorganize the bins and straighten up the mess, you decide to mention the mess issue to Bill in your afternoon meeting.

You describe the bin mess and using the incorrect length dowels to Bill. He is glad you mentioned the problem to him. You are not the first person to mention similar issues with tools and parts not being put away correctly. Chances are there is a bigger safety issue here that needs to be addressed in the next staff meeting.

In any workplace, following proper safety and cleanup procedures is always important. This is especially crucial in manufacturing where people are constantly working with heavy, costly and sometimes dangerous equipment. When issues and problems arise, it is important that they are addressed in an efficient and timely manner. Effective communication is an important tool because it can prevent problems from recurring, avoid injury to personnel, reduce rework and scrap, and ultimately, reduce cost, and save money.

You now know that the end mill was damaged when it hit the dowel pin. It seems to you that the easiest thing to do would be to edit the G-code and raise the Z-axis position of the spindle before it returns to the home position. You open the CNC control program and edit the G-code, raising the Z-axis. Starting over, you follow the setup sheet and re-setup the job. This time, you use the CNC single step/single block function, to do another dry run of the part. Everything looks good, so you run the job again and create the first part.

At the end of the day, you are reviewing your progress with your trainer, Bill. After you describe the day's events, he reminds you to always think about safety and the importance of following work procedures. He decides to bring the issue up in the next morning meeting as a reminder to everyone.

In any workplace, following proper procedures (especially those that involve safety) is always important. This is especially crucial in manufacturing where people are constantly working with heavy, costly, and sometimes dangerous equipment. When issues and problems arise, it is important that they are addressed in an efficient and timely manner. Effective communication is an important tool because it can prevent problems from recurring, avoid injury to personnel, reduce rework and scrap, and ultimately, reduce cost, and save money. One tool to improve communication is the morning meeting or huddle.

The next morning, you check the G-code to determine what is wrong with the coolant. You notice that the coolant is turned off at the beginning of the code and also at the end of the code. This is strange. You change the G-code to turn the coolant on at the beginning of the run and off at the end. This works and you create the rest of the parts.

Throughout the day, you keep wondering what caused the G-code error. At lunch, you mention the G-code error to your coworker, John. John is not surprised. He said that he encountered a similar problem earlier this week. You decide to talk with your supervisor the next time you see him.

You are in luck. You see your supervisor by the door getting ready to leave. You hurry over to talk with him. You start off by telling him about how you asked Bill for help. Then you tell him there was a problem and the end mill was damaged. You describe the coolant problem in the G-code. Oh, and by the way, John has seen a similar problem before.

Your supervisor doesn't seem overly concerned, errors happen. He tells you "Good job, I am glad you were able to fix the issue." You are not sure whether your supervisor understood your explanation of what happened or that it had happened before.

The challenge of communicating in the workplace is learning how to share your ideas and concerns. If you need to tell your supervisor that something is not going well, it is important to remember that timing, preparation, and attitude are extremely important.

It is the end of your shift, but you want to let the next shift know that the coolant didn't turn on. You do not see your trainer or supervisor around. You decide to leave a note for the next shift so they are aware of the possible coolant problem. You write a sticky note and leave it on the monitor of the CNC control system.

How effective do you think this solution was? Did it address the problem?

In this scenario, you discovered several problems with the G-code that need to be addressed. When issues and problems arise, it is important that they are addressed in an efficient and timely manner. Effective communication is an important tool because it can prevent problems from recurring and avoid injury to personnel. The challenge of communicating in the workplace is learning how and when to share your ideas and concerns. If you need to tell your co-workers or supervisor that there is a problem, it is important to remember that timing and the method of communication are extremely important.

You are able to fix the coolant problem in the G-code. While you are glad that the problem is fixed, you are worried about why it happened in the first place. It is important to remember that if a problem keeps reappearing, you may not be fixing the right problem. You may only be addressing the symptoms.

You decide to talk to your trainer. Bill is glad you mentioned the problem to him. You are the third worker to mention G-code issues over the last week. You noticed the coolant problems in your G-code, John noticed a Z-axis issue in his G-code, and Sam had issues with both the Z-axis and the coolant. Chances are, there is a bigger problem and Bill will need to investigate the root cause .

Over lunch, you ask your coworkers about the G-code problem and what may be causing the error. Several people mention having similar problems but do not know the cause.

You have now talked to three coworkers who have all experienced similar coolant G-code problems. You make a list of who had the problem, when they had the problem, and what each person told you.

When you see your supervisor later that afternoon, you are ready to talk with him. You describe the problem you had with your component and the damaged bit. You then go on to tell him about talking with Bill and discovering the G-code issue. You show him your notes on your coworkers' coolant issues, and explain that you think there might be a bigger problem.

You supervisor thanks you for your initiative in identifying this problem. It sounds like there is a bigger problem and he will need to investigate the root cause. He decides to call a team huddle to discuss the issue, gather more information, and talk with the team about the importance of communication.

Root Cause Analysis

Root cause analysis ( RCA ) is a method of problem solving that identifies the underlying causes of an issue. Root cause analysis helps people answer the question of why the problem occurred in the first place. RCA uses clear cut steps in its associated tools, like the "5 Whys Analysis" and the "Cause and Effect Diagram," to identify the origin of the problem, so that you can:

Determine what happened.
Determine why it happened.
Fix the problem so it won’t happen again.

RCA works under the idea that systems and events are connected. An action in one area triggers an action in another, and another, and so on. By tracing back these actions, you can discover where the problem started and how it developed into the problem you're now facing. Root cause analysis can prevent problems from recurring, reduce injury to personnel, reduce rework and scrap, and ultimately, reduce cost and save money. There are many different RCA techniques available to determine the root cause of a problem. These are just a few:

Root Cause Analysis Tools
5 Whys Analysis
Fishbone or Cause and Effect Diagram
Pareto Analysis

How Huddles Work

Communication is a vital part of any setting where people work together. Effective communication helps employees and managers form efficient teams. It builds trusts between employees and management, and reduces unnecessary competition because each employee knows how their part fits in the larger goal.

One tool that management can use to promote communication in the workplace is the huddle . Just like football players on the field, a huddle is a short meeting where everyone is standing in a circle. A daily team huddle ensures that team members are aware of changes to the schedule, reiterated problems and safety issues, and how their work impacts one another. When done right, huddles create collaboration, communication, and accountability to results. Impromptu huddles can be used to gather information on a specific issue and get each team member's input.

The most important thing to remember about huddles is that they are short, lasting no more than 10 minutes, and their purpose is to communicate and identify. In essence, a huddle’s purpose is to identify priorities, communicate essential information, and discover roadblocks to productivity.

Who uses huddles? Many industries and companies use daily huddles. At first thought, most people probably think of hospitals and their daily patient update meetings, but lots of managers use daily meetings to engage their employees. Here are a few examples:

Brian Scudamore, CEO of 1-800-Got-Junk? , uses the daily huddle as an operational tool to take the pulse of his employees and as a motivational tool. Watch a morning huddle meeting .
Fusion OEM, an outsourced manufacturing and production company. What do employees take away from the daily huddle meeting .
Biz-Group, a performance consulting group. Tips for a successful huddle .

Brainstorming

One tool that can be useful in problem solving is brainstorming . Brainstorming is a creativity technique designed to generate a large number of ideas for the solution to a problem. The method was first popularized in 1953 by Alex Faickney Osborn in the book Applied Imagination . The goal is to come up with as many ideas as you can in a fixed amount of time. Although brainstorming is best done in a group, it can be done individually. Like most problem solving techniques, brainstorming is a process.

Define a clear objective.
Have an agreed a time limit.
During the brainstorming session, write down everything that comes to mind, even if the idea sounds crazy.
If one idea leads to another, write down that idea too.
Combine and refine ideas into categories of solutions.
Assess and analyze each idea as a potential solution.

When used during problem solving, brainstorming can offer companies new ways of encouraging staff to think creatively and improve production. Brainstorming relies on team members' diverse experiences, adding to the richness of ideas explored. This means that you often find better solutions to the problems. Team members often welcome the opportunity to contribute ideas and can provide buy-in for the solution chosen—after all, they are more likely to be committed to an approach if they were involved in its development. What's more, because brainstorming is fun, it helps team members bond.

Watch Peggy Morgan Collins, a marketing executive at Power Curve Communications discuss How to Stimulate Effective Brainstorming .
Watch Kim Obbink, CEO of Filter Digital, a digital content company, and her team share their top five rules for How to Effectively Generate Ideas .

Importance of Good Communication and Problem Description

talking too much when describing a problem

Communication is one of the most frequent activities we engage in on a day-to-day basis. At some point, we have all felt that we did not effectively communicate an idea as we would have liked. The key to effective communication is preparation. Rather than attempting to haphazardly improvise something, take a few minutes and think about what you want say and how you will say it. If necessary, write yourself a note with the key points or ideas in the order you want to discuss them. The notes can act as a reminder or guide when you talk to your supervisor.

Tips for clear communication of an issue:

Provide a clear summary of your problem. Start at the beginning, give relevant facts, timelines, and examples.
Avoid including your opinion or personal attacks in your explanation.
Avoid using words like "always" or "never," which can give the impression that you are exaggerating the problem.
If this is an ongoing problem and you have collected documentation, give it to your supervisor once you have finished describing the problem.
Remember to listen to what's said in return; communication is a two-way process.

Not all communication is spoken. Body language is nonverbal communication that includes your posture, your hands and whether you make eye contact. These gestures can be subtle or overt, but most importantly they communicate meaning beyond what is said. When having a conversation, pay attention to how you stand. A stiff position with arms crossed over your chest may imply that you are being defensive even if your words state otherwise. Shoving your hands in your pockets when speaking could imply that you have something to hide. Be wary of using too many hand gestures because this could distract listeners from your message.

The challenge of communicating in the workplace is learning how and when to share your ideas or concerns. If you need to tell your supervisor or co-worker about something that is not going well, keep in mind that good timing and good attitude will go a long way toward helping your case.

Like all skills, effective communication needs to be practiced. Toastmasters International is perhaps the best known public speaking organization in the world. Toastmasters is open to anyone who wish to improve their speaking skills and is willing to put in the time and effort to do so. To learn more, visit Toastmasters International .

Methods of Communication

Communication of problems and issues in any workplace is important, particularly when safety is involved. It is therefore crucial in manufacturing where people are constantly working with heavy, costly, and sometimes dangerous equipment. As issues and problems arise, they need to be addressed in an efficient and timely manner. Effective communication is an important skill because it can prevent problems from recurring, avoid injury to personnel, reduce rework and scrap, and ultimately, reduce cost and save money.

There are many different ways to communicate: in person, by phone, via email, or written. There is no single method that fits all communication needs, each one has its time and place.

In person: In the workplace, face-to-face meetings should be utilized whenever possible. Being able to see the person you need to speak to face-to-face gives you instant feedback and helps you gauge their response through their body language. Be careful of getting sidetracked in conversation when you need to communicate a problem.

Email: Email has become the communication standard for most businesses. It can be accessed from almost anywhere and is great for things that don’t require an immediate response. Email is a great way to communicate non-urgent items to large amounts of people or just your team members. One thing to remember is that most people's inboxes are flooded with emails every day and unless they are hyper vigilant about checking everything, important items could be missed. For issues that are urgent, especially those around safety, email is not always be the best solution.

Phone: Phone calls are more personal and direct than email. They allow us to communicate in real time with another person, no matter where they are. Not only can talking prevent miscommunication, it promotes a two-way dialogue. You don’t have to worry about your words being altered or the message arriving on time. However, mobile phone use and the workplace don't always mix. In particular, using mobile phones in a manufacturing setting can lead to a variety of problems, cause distractions, and lead to serious injury.

Written: Written communication is appropriate when detailed instructions are required, when something needs to be documented, or when the person is too far away to easily speak with over the phone or in person.

There is no "right" way to communicate, but you should be aware of how and when to use the appropriate form of communication for your situation. When deciding the best way to communicate with a co-worker or manager, put yourself in their shoes, and think about how you would want to learn about the issue. Also, consider what information you would need to know to better understand the issue. Use your good judgment of the situation and be considerate of your listener's viewpoint.

Did you notice any other potential problems in the previous exercise?

[Page 6:] Did you notice any other potential problems in the previous exercise?

Summary of Strategies

In this exercise, you were given a scenario in which there was a problem with a component you were creating on a CNC machine. You were then asked how you wanted to proceed. Depending on your path through this exercise, you might have found an easy solution and fixed it yourself, asked for help and worked with your trainer, or discovered an ongoing G-code problem that was bigger than you initially thought.

When issues and problems arise, it is important that they are addressed in an efficient and timely manner. Communication is an important tool because it can prevent problems from recurring, avoid injury to personnel, reduce rework and scrap, and ultimately, reduce cost, and save money. Although, each path in this exercise ended with a description of a problem solving tool for your toolbox, the first step is always to identify the problem and define the context in which it happened.

There are several strategies that can be used to identify the root cause of a problem. Root cause analysis (RCA) is a method of problem solving that helps people answer the question of why the problem occurred. RCA uses a specific set of steps, with associated tools like the “5 Why Analysis" or the “Cause and Effect Diagram,” to identify the origin of the problem, so that you can:

Once the underlying cause is identified and the scope of the issue defined, the next step is to explore possible strategies to fix the problem.

If you are not sure how to fix the problem, it is okay to ask for help. Problem solving is a process and a skill that is learned with practice. It is important to remember that everyone makes mistakes and that no one knows everything. Life is about learning. It is okay to ask for help when you don’t have the answer. When you collaborate to solve problems you improve workplace communication and accelerates finding solutions as similar problems arise.

One tool that can be useful for generating possible solutions is brainstorming . Brainstorming is a technique designed to generate a large number of ideas for the solution to a problem. The method was first popularized in 1953 by Alex Faickney Osborn in the book Applied Imagination. The goal is to come up with as many ideas as you can, in a fixed amount of time. Although brainstorming is best done in a group, it can be done individually.

Depending on your path through the exercise, you may have discovered that a couple of your coworkers had experienced similar problems. This should have been an indicator that there was a larger problem that needed to be addressed.

In any workplace, communication of problems and issues (especially those that involve safety) is always important. This is especially crucial in manufacturing where people are constantly working with heavy, costly, and sometimes dangerous equipment. When issues and problems arise, it is important that they be addressed in an efficient and timely manner. Effective communication is an important tool because it can prevent problems from recurring, avoid injury to personnel, reduce rework and scrap, and ultimately, reduce cost and save money.

One strategy for improving communication is the huddle . Just like football players on the field, a huddle is a short meeting with everyone standing in a circle. A daily team huddle is a great way to ensure that team members are aware of changes to the schedule, any problems or safety issues are identified and that team members are aware of how their work impacts one another. When done right, huddles create collaboration, communication, and accountability to results. Impromptu huddles can be used to gather information on a specific issue and get each team member's input.

To learn more about different problem solving strategies, choose an option below. These strategies accompany the outcomes of different decision paths in the problem solving exercise.

View Problem Solving Strategies Select a strategy below... Root Cause Analysis How Huddles Work Brainstorming Importance of Good Problem Description Methods of Communication

Provide a clear summary of the problem. Start at the beginning, give relevant facts, timelines, and examples.

In person: In the workplace, face-to-face meetings should be utilized whenever possible. Being able to see the person you need to speak to face-to-face gives you instant feedback and helps you gauge their response in their body language. Be careful of getting sidetracked in conversation when you need to communicate a problem.

There is no "right" way to communicate, but you should be aware of how and when to use the appropriate form of communication for the situation. When deciding the best way to communicate with a co-worker or manager, put yourself in their shoes, and think about how you would want to learn about the issue. Also, consider what information you would need to know to better understand the issue. Use your good judgment of the situation and be considerate of your listener's viewpoint.

"Never try to solve all the problems at once — make them line up for you one-by-one.” — Richard Sloma

Problem Solving: An Important Job Skill

Problem solving improves efficiency and communication on the shop floor. It increases a company's efficiency and profitability, so it's one of the top skills employers look for when hiring new employees. Recent industry surveys show that employers consider soft skills, such as problem solving, as critical to their business’s success.

The 2011 survey, "Boiling Point? The skills gap in U.S. manufacturing ," polled over a thousand manufacturing executives who reported that the number one skill deficiency among their current employees is problem solving, which makes it difficult for their companies to adapt to the changing needs of the industry.

In this video, industry professionals discuss their expectations and present tips for new employees joining the manufacturing workforce.

Quick Summary

[Quick Summary: Question1] What are two things you learned in this case study?
What question(s) do you still have about the case study?
[Quick Summary: Question2] What question(s) do you still have about the case study?
Is there anything you would like to learn more about with respect to this case study?
[Quick Summary: Question3] Is there anything you would like to learn more about with respect to this case study?

Join Mind Tools

How Good Is Your Problem Solving?

Use a systematic approach.

Good problem solving skills are fundamentally important if you're going to be successful in your career.

But problems are something that we don't particularly like.

They're time-consuming.

They muscle their way into already packed schedules.

They force us to think about an uncertain future.

And they never seem to go away!

That's why, when faced with problems, most of us try to eliminate them as quickly as possible. But have you ever chosen the easiest or most obvious solution – and then realized that you have entirely missed a much better solution? Or have you found yourself fixing just the symptoms of a problem, only for the situation to get much worse?

To be an effective problem-solver, you need to be systematic and logical in your approach. This quiz helps you assess your current approach to problem solving. By improving this, you'll make better overall decisions. And as you increase your confidence with solving problems, you'll be less likely to rush to the first solution – which may not necessarily be the best one.

Once you've completed the quiz, we'll direct you to tools and resources that can help you make the most of your problem-solving skills.

How Good Are You at Solving Problems?

Instructions.

For each statement, click the button in the column that best describes you. Please answer questions as you actually are (rather than how you think you should be), and don't worry if some questions seem to score in the 'wrong direction'. When you are finished, please click the 'Calculate My Total' button at the bottom of the test.

Your last quiz results are shown.

You last completed this quiz on , at .

Score Interpretation

Answering these questions should have helped you recognize the key steps associated with effective problem solving.

This quiz is based on Dr Min Basadur's Simplexity Thinking problem-solving model. This eight-step process follows the circular pattern shown below, within which current problems are solved and new problems are identified on an ongoing basis. This assessment has not been validated and is intended for illustrative purposes only.

Figure 1 – The Simplexity Thinking Process

Below, we outline the tools and strategies you can use for each stage of the problem-solving process. Enjoy exploring these stages!

Step 1: Find the Problem

(Questions 7, 12)

Some problems are very obvious, however others are not so easily identified. As part of an effective problem-solving process, you need to look actively for problems – even when things seem to be running fine. Proactive problem solving helps you avoid emergencies and allows you to be calm and in control when issues arise.

These techniques can help you do this:

PEST Analysis helps you pick up changes to your environment that you should be paying attention to. Make sure too that you're watching changes in customer needs and market dynamics, and that you're monitoring trends that are relevant to your industry.
Risk Analysis helps you identify significant business risks.
Failure Modes and Effects Analysis helps you identify possible points of failure in your business process, so that you can fix these before problems arise.
After Action Reviews help you scan recent performance to identify things that can be done better in the future.
Where you have several problems to solve, our articles on Prioritization and Pareto Analysis help you think about which ones you should focus on first.

Step 2: Find the Facts

(Questions 10, 14)

After identifying a potential problem, you need information. What factors contribute to the problem? Who is involved with it? What solutions have been tried before? What do others think about the problem?

If you move forward to find a solution too quickly, you risk relying on imperfect information that's based on assumptions and limited perspectives, so make sure that you research the problem thoroughly.

Step 3: Define the Problem

(Questions 3, 9)

Now that you understand the problem, define it clearly and completely. Writing a clear problem definition forces you to establish specific boundaries for the problem. This keeps the scope from growing too large, and it helps you stay focused on the main issues.

A great tool to use at this stage is CATWOE . With this process, you analyze potential problems by looking at them from six perspectives, those of its Customers; Actors (people within the organization); the Transformation, or business process; the World-view, or top-down view of what's going on; the Owner; and the wider organizational Environment. By looking at a situation from these perspectives, you can open your mind and come to a much sharper and more comprehensive definition of the problem.

Cause and Effect Analysis is another good tool to use here, as it helps you think about the many different factors that can contribute to a problem. This helps you separate the symptoms of a problem from its fundamental causes.

Step 4: Find Ideas

(Questions 4, 13)

With a clear problem definition, start generating ideas for a solution. The key here is to be flexible in the way you approach a problem. You want to be able to see it from as many perspectives as possible. Looking for patterns or common elements in different parts of the problem can sometimes help. You can also use metaphors and analogies to help analyze the problem, discover similarities to other issues, and think of solutions based on those similarities.

Traditional brainstorming and reverse brainstorming are very useful here. By taking the time to generate a range of creative solutions to the problem, you'll significantly increase the likelihood that you'll find the best possible solution, not just a semi-adequate one. Where appropriate, involve people with different viewpoints to expand the volume of ideas generated.

Don't evaluate your ideas until step 5. If you do, this will limit your creativity at too early a stage.

Step 5: Select and Evaluate

(Questions 6, 15)

After finding ideas, you'll have many options that must be evaluated. It's tempting at this stage to charge in and start discarding ideas immediately. However, if you do this without first determining the criteria for a good solution, you risk rejecting an alternative that has real potential.

Decide what elements are needed for a realistic and practical solution, and think about the criteria you'll use to choose between potential solutions.

Paired Comparison Analysis , Decision Matrix Analysis and Risk Analysis are useful techniques here, as are many of the specialist resources available within our Decision-Making section . Enjoy exploring these!

Step 6: Plan

(Questions 1, 16)

You might think that choosing a solution is the end of a problem-solving process. In fact, it's simply the start of the next phase in problem solving: implementation. This involves lots of planning and preparation. If you haven't already developed a full Risk Analysis in the evaluation phase, do so now. It's important to know what to be prepared for as you begin to roll out your proposed solution.

The type of planning that you need to do depends on the size of the implementation project that you need to set up. For small projects, all you'll often need are Action Plans that outline who will do what, when, and how. Larger projects need more sophisticated approaches – you'll find out more about these in the Mind Tools Project Management section. And for projects that affect many other people, you'll need to think about Change Management as well.

Here, it can be useful to conduct an Impact Analysis to help you identify potential resistance as well as alert you to problems you may not have anticipated. Force Field Analysis will also help you uncover the various pressures for and against your proposed solution. Once you've done the detailed planning, it can also be useful at this stage to make a final Go/No-Go Decision , making sure that it's actually worth going ahead with the selected option.

Step 7: Sell the Idea

(Questions 5, 8)

As part of the planning process, you must convince other stakeholders that your solution is the best one. You'll likely meet with resistance, so before you try to “sell” your idea, make sure you've considered all the consequences.

As you begin communicating your plan, listen to what people say, and make changes as necessary. The better the overall solution meets everyone's needs, the greater its positive impact will be! For more tips on selling your idea, read our article on Creating a Value Proposition and use our Sell Your Idea Bite-Sized Training session.

Step 8: Act

(Questions 2, 11)

Finally, once you've convinced your key stakeholders that your proposed solution is worth running with, you can move on to the implementation stage. This is the exciting and rewarding part of problem solving, which makes the whole process seem worthwhile.

This action stage is an end, but it's also a beginning: once you've completed your implementation, it's time to move into the next cycle of problem solving by returning to the scanning stage. By doing this, you'll continue improving your organization as you move into the future.

Problem solving is an exceptionally important workplace skill.

Being a competent and confident problem solver will create many opportunities for you. By using a well-developed model like Simplexity Thinking for solving problems, you can approach the process systematically, and be comfortable that the decisions you make are solid.

Given the unpredictable nature of problems, it's very reassuring to know that, by following a structured plan, you've done everything you can to resolve the problem to the best of your ability.

This site teaches you the skills you need for a happy and successful career; and this is just one of many tools and resources that you'll find here at Mind Tools. Subscribe to our free newsletter , or join the Mind Tools Club and really supercharge your career!

Rate this resource

The Mind Tools Club gives you exclusive tips and tools to boost your career - plus a friendly community and support from our career coaches!

problem solving ability test introduction

Comments (220)

Over a month ago Sonia_H wrote Hi PANGGA, This is great news! Thanks for sharing your experience. We hope these 8 steps outlined will help you in multiple ways. ~Sonia Mind Tools Coach
Over a month ago PANGGA wrote Thank you for this mind tool. I got to know my skills in solving problem. It will serve as my guide on facing and solving problem that I might encounter.
Over a month ago Sarah_H wrote Wow, thanks for your very detailed feedback HardipG. The Mind Tools team will take a look at your feedback and suggestions for improvement. Best wishes, Sarah Mind Tools Coach

Please wait...

Brain Development
Childhood & Adolescence
Diet & Lifestyle
Emotions, Stress & Anxiety
Learning & Memory
Thinking & Awareness
Alzheimer's & Dementia
Childhood Disorders
Immune System Disorders
Mental Health
Neurodegenerative Disorders
Infectious Disease
Neurological Disorders A-Z
Body Systems
Cells & Circuits
Genes & Molecules
The Arts & the Brain
Law, Economics & Ethics
Neuroscience in the News
Supporting Research
Tech & the Brain
Animals in Research
BRAIN Initiative
Meet the Researcher
Neuro-technologies
Tools & Techniques
Core Concepts
For Educators
Ask an Expert
The Brain Facts Book

Test Your Problem-Solving Skills

Personalize your emails.

Personalize your monthly updates from BrainFacts.org by choosing the topics that you care about most!

Find a Neuroscientist

Engage local scientists to educate your community about the brain.

Image of the Week

Check out the Image of the Week Archive.

SUPPORTING PARTNERS

Privacy Policy
Accessibility Policy
Terms and Conditions
Manage Cookies

Some pages on this website provide links that require Adobe Reader to view.

How it works

For Business

Join Mind Tools

Self-Assessment • 20 min read

How Good Is Your Problem Solving?

Use a systematic approach..

By the Mind Tools Content Team

Good problem solving skills are fundamentally important if you're going to be successful in your career.

But problems are something that we don't particularly like.

They're time-consuming.

They muscle their way into already packed schedules.

They force us to think about an uncertain future.

And they never seem to go away!

Once you've completed the quiz, we'll direct you to tools and resources that can help you make the most of your problem-solving skills.

How Good Are You at Solving Problems?

Instructions.

Answering these questions should have helped you recognize the key steps associated with effective problem solving.

Below, we outline the tools and strategies you can use for each stage of the problem-solving process. Enjoy exploring these stages!

Step 1: Find the Problem (Questions 7, 12)

These techniques can help you do this:

PEST Analysis helps you pick up changes to your environment that you should be paying attention to. Make sure too that you're watching changes in customer needs and market dynamics, and that you're monitoring trends that are relevant to your industry.

Risk Analysis helps you identify significant business risks.

Failure Modes and Effects Analysis helps you identify possible points of failure in your business process, so that you can fix these before problems arise.

After Action Reviews help you scan recent performance to identify things that can be done better in the future.

Where you have several problems to solve, our articles on Prioritization and Pareto Analysis help you think about which ones you should focus on first.

Step 2: Find the Facts (Questions 10, 14)

Step 3: Define the Problem (Questions 3, 9)

Step 4: Find Ideas (Questions 4, 13)

Tip: Don't evaluate your ideas until step 5. If you do, this will limit your creativity at too early a stage.

Step 5: Select and Evaluate (Questions 6, 15)

Decide what elements are needed for a realistic and practical solution, and think about the criteria you'll use to choose between potential solutions.

Step 6: Plan (Questions 1, 16)

The type of planning that you need to do depends on the size of the implementation project that you need to set up. For small projects, all you'll often need are Action Plans that outline who will do what, when, and how. Larger projects need more sophisticated approaches – you'll find out more about these in the article What is Project Management? And for projects that affect many other people, you'll need to think about Change Management as well.

Step 7: Sell the Idea (Questions 5, 8)

Step 8: Act (Questions 2, 11)

Problem solving is an exceptionally important workplace skill.

Given the unpredictable nature of problems, it's very reassuring to know that, by following a structured plan, you've done everything you can to resolve the problem to the best of your ability.

This assessment has not been validated and is intended for illustrative purposes only. It is just one of many Mind Tool quizzes that can help you to evaluate your abilities in a wide range of important career skills.

If you want to reproduce this quiz, you can purchase downloadable copies in our Store .

You've accessed 1 of your 2 free resources.

Get unlimited access

Discover more content

4 logical fallacies.

Avoid Common Types of Faulty Reasoning

Problem Solving

Add comment

Comments (2)

Afkar Hashmi

😇 This tool is very useful for me.

over 1 year

Very impactful

Get 20% off your first year of Mind Tools

Our on-demand e-learning resources let you learn at your own pace, fitting seamlessly into your busy workday. Join today and save with our limited time offer!

Sign-up to our newsletter

Subscribing to the Mind Tools newsletter will keep you up-to-date with our latest updates and newest resources.

Subscribe now

Business Skills

Personal Development

Leadership and Management

Discover something new today

How do i manage a hybrid team.

Adjusting your management style to a hybrid world

The Life Career Rainbow

Finding a Work-Life Balance That Suits You

How Emotionally Intelligent Are You?

Boosting Your People Skills

Self-Assessment

What's Your Leadership Style?

Learn About the Strengths and Weaknesses of the Way You Like to Lead

Recommended for you

Get out of your own way at work: and help others to do the same.

Mark Goulston

Book Insights

Business Operations and Process Management

Strategy Tools

Customer Service

Business Ethics and Values

Handling Information and Data

Project Management

Knowledge Management

Self-Development and Goal Setting

Time Management

Presentation Skills

Learning Skills

Career Skills

Communication Skills

Negotiation, Persuasion and Influence

Working With Others

Difficult Conversations

Creativity Tools

Self-Management

Work-Life Balance

Stress Management and Wellbeing

Coaching and Mentoring

Change Management

Team Management

Managing Conflict

Delegation and Empowerment

Performance Management

Leadership Skills

Developing Your Team

Talent Management

Decision Making

Pain Points

Creative Problem-Solving Test

Do you typically approach a problem from many perspectives or opt for the same old solution that worked in the past? In his work on human motivation, Robert E. Franken states that in order to be creative, you need to be able to view things from different perspectives.

Creativity is linked to fundamental qualities of thinking, such as flexibility and tolerance of ambiguity. This Creative Problem-solving Test was developed to evaluate whether your attitude towards problem-solving and the manner in which you approach a problem are conducive to creative thinking.

This test is made up of two types of questions: scenarios and self-assessment. For each scenario, answer according to how you would most likely behave in a similar situation. For the self-assessment questions, indicate the degree to which the given statements apply to you. In order to receive the most accurate results, please answer each question as honestly as possible.

After finishing this test you will receive a FREE snapshot report with a summary evaluation and graph. You will then have the option to purchase the full results for $6.95

This test is intended for informational and entertainment purposes only. It is not a substitute for professional diagnosis or for the treatment of any health condition. If you would like to seek the advice of a licensed mental health professional you can search Psychology Today's directory here .

Find a Therapist
Find a Treatment Center
Find a Psychiatrist
Find a Support Group
Find Teletherapy
United States
Brooklyn, NY
Chicago, IL
Houston, TX
Los Angeles, CA
New York, NY
Portland, OR
San Diego, CA
San Francisco, CA
Seattle, WA
Washington, DC
Asperger's
Bipolar Disorder
Chronic Pain
Eating Disorders
Passive Aggression
Personality
Goal Setting
Positive Psychology
Stopping Smoking
Low Sexual Desire
Relationships
Child Development
Therapy Center NEW
Diagnosis Dictionary
Types of Therapy

Understanding what emotional intelligence looks like and the steps needed to improve it could light a path to a more emotionally adept world.

Coronavirus Disease 2019
Affective Forecasting
Neuroscience

7.3 Problem Solving

Learning objectives.

Describe problem solving strategies
Define algorithm and heuristic
Explain some common roadblocks to effective problem solving

People face problems every day—usually, multiple problems throughout the day. Sometimes these problems are straightforward: To double a recipe for pizza dough, for example, all that is required is that each ingredient in the recipe be doubled. Sometimes, however, the problems we encounter are more complex. For example, say you have a work deadline, and you must mail a printed copy of a report to your supervisor by the end of the business day. The report is time-sensitive and must be sent overnight. You finished the report last night, but your printer will not work today. What should you do? First, you need to identify the problem and then apply a strategy for solving the problem.

Problem-Solving Strategies

When you are presented with a problem—whether it is a complex mathematical problem or a broken printer, how do you solve it? Before finding a solution to the problem, the problem must first be clearly identified. After that, one of many problem solving strategies can be applied, hopefully resulting in a solution.

A problem-solving strategy is a plan of action used to find a solution. Different strategies have different action plans associated with them ( Table 7.2 ). For example, a well-known strategy is trial and error . The old adage, “If at first you don’t succeed, try, try again” describes trial and error. In terms of your broken printer, you could try checking the ink levels, and if that doesn’t work, you could check to make sure the paper tray isn’t jammed. Or maybe the printer isn’t actually connected to your laptop. When using trial and error, you would continue to try different solutions until you solved your problem. Although trial and error is not typically one of the most time-efficient strategies, it is a commonly used one.

Another type of strategy is an algorithm. An algorithm is a problem-solving formula that provides you with step-by-step instructions used to achieve a desired outcome (Kahneman, 2011). You can think of an algorithm as a recipe with highly detailed instructions that produce the same result every time they are performed. Algorithms are used frequently in our everyday lives, especially in computer science. When you run a search on the Internet, search engines like Google use algorithms to decide which entries will appear first in your list of results. Facebook also uses algorithms to decide which posts to display on your newsfeed. Can you identify other situations in which algorithms are used?

A heuristic is another type of problem solving strategy. While an algorithm must be followed exactly to produce a correct result, a heuristic is a general problem-solving framework (Tversky & Kahneman, 1974). You can think of these as mental shortcuts that are used to solve problems. A “rule of thumb” is an example of a heuristic. Such a rule saves the person time and energy when making a decision, but despite its time-saving characteristics, it is not always the best method for making a rational decision. Different types of heuristics are used in different types of situations, but the impulse to use a heuristic occurs when one of five conditions is met (Pratkanis, 1989):

When one is faced with too much information
When the time to make a decision is limited
When the decision to be made is unimportant
When there is access to very little information to use in making the decision
When an appropriate heuristic happens to come to mind in the same moment

Working backwards is a useful heuristic in which you begin solving the problem by focusing on the end result. Consider this example: You live in Washington, D.C. and have been invited to a wedding at 4 PM on Saturday in Philadelphia. Knowing that Interstate 95 tends to back up any day of the week, you need to plan your route and time your departure accordingly. If you want to be at the wedding service by 3:30 PM, and it takes 2.5 hours to get to Philadelphia without traffic, what time should you leave your house? You use the working backwards heuristic to plan the events of your day on a regular basis, probably without even thinking about it.

Another useful heuristic is the practice of accomplishing a large goal or task by breaking it into a series of smaller steps. Students often use this common method to complete a large research project or long essay for school. For example, students typically brainstorm, develop a thesis or main topic, research the chosen topic, organize their information into an outline, write a rough draft, revise and edit the rough draft, develop a final draft, organize the references list, and proofread their work before turning in the project. The large task becomes less overwhelming when it is broken down into a series of small steps.

Everyday Connection

Solving puzzles.

Problem-solving abilities can improve with practice. Many people challenge themselves every day with puzzles and other mental exercises to sharpen their problem-solving skills. Sudoku puzzles appear daily in most newspapers. Typically, a sudoku puzzle is a 9×9 grid. The simple sudoku below ( Figure 7.8 ) is a 4×4 grid. To solve the puzzle, fill in the empty boxes with a single digit: 1, 2, 3, or 4. Here are the rules: The numbers must total 10 in each bolded box, each row, and each column; however, each digit can only appear once in a bolded box, row, and column. Time yourself as you solve this puzzle and compare your time with a classmate.

Here is another popular type of puzzle ( Figure 7.9 ) that challenges your spatial reasoning skills. Connect all nine dots with four connecting straight lines without lifting your pencil from the paper:

Take a look at the “Puzzling Scales” logic puzzle below ( Figure 7.10 ). Sam Loyd, a well-known puzzle master, created and refined countless puzzles throughout his lifetime (Cyclopedia of Puzzles, n.d.).

Pitfalls to Problem Solving

Not all problems are successfully solved, however. What challenges stop us from successfully solving a problem? Albert Einstein once said, “Insanity is doing the same thing over and over again and expecting a different result.” Imagine a person in a room that has four doorways. One doorway that has always been open in the past is now locked. The person, accustomed to exiting the room by that particular doorway, keeps trying to get out through the same doorway even though the other three doorways are open. The person is stuck—but she just needs to go to another doorway, instead of trying to get out through the locked doorway. A mental set is where you persist in approaching a problem in a way that has worked in the past but is clearly not working now.

Functional fixedness is a type of mental set where you cannot perceive an object being used for something other than what it was designed for. During the Apollo 13 mission to the moon, NASA engineers at Mission Control had to overcome functional fixedness to save the lives of the astronauts aboard the spacecraft. An explosion in a module of the spacecraft damaged multiple systems. The astronauts were in danger of being poisoned by rising levels of carbon dioxide because of problems with the carbon dioxide filters. The engineers found a way for the astronauts to use spare plastic bags, tape, and air hoses to create a makeshift air filter, which saved the lives of the astronauts.

Link to Learning

Check out this Apollo 13 scene where the group of NASA engineers are given the task of overcoming functional fixedness.

Researchers have investigated whether functional fixedness is affected by culture. In one experiment, individuals from the Shuar group in Ecuador were asked to use an object for a purpose other than that for which the object was originally intended. For example, the participants were told a story about a bear and a rabbit that were separated by a river and asked to select among various objects, including a spoon, a cup, erasers, and so on, to help the animals. The spoon was the only object long enough to span the imaginary river, but if the spoon was presented in a way that reflected its normal usage, it took participants longer to choose the spoon to solve the problem. (German & Barrett, 2005). The researchers wanted to know if exposure to highly specialized tools, as occurs with individuals in industrialized nations, affects their ability to transcend functional fixedness. It was determined that functional fixedness is experienced in both industrialized and nonindustrialized cultures (German & Barrett, 2005).

In order to make good decisions, we use our knowledge and our reasoning. Often, this knowledge and reasoning is sound and solid. Sometimes, however, we are swayed by biases or by others manipulating a situation. For example, let’s say you and three friends wanted to rent a house and had a combined target budget of $1,600. The realtor shows you only very run-down houses for $1,600 and then shows you a very nice house for $2,000. Might you ask each person to pay more in rent to get the $2,000 home? Why would the realtor show you the run-down houses and the nice house? The realtor may be challenging your anchoring bias. An anchoring bias occurs when you focus on one piece of information when making a decision or solving a problem. In this case, you’re so focused on the amount of money you are willing to spend that you may not recognize what kinds of houses are available at that price point.

The confirmation bias is the tendency to focus on information that confirms your existing beliefs. For example, if you think that your professor is not very nice, you notice all of the instances of rude behavior exhibited by the professor while ignoring the countless pleasant interactions he is involved in on a daily basis. Hindsight bias leads you to believe that the event you just experienced was predictable, even though it really wasn’t. In other words, you knew all along that things would turn out the way they did. Representative bias describes a faulty way of thinking, in which you unintentionally stereotype someone or something; for example, you may assume that your professors spend their free time reading books and engaging in intellectual conversation, because the idea of them spending their time playing volleyball or visiting an amusement park does not fit in with your stereotypes of professors.

Finally, the availability heuristic is a heuristic in which you make a decision based on an example, information, or recent experience that is that readily available to you, even though it may not be the best example to inform your decision . Biases tend to “preserve that which is already established—to maintain our preexisting knowledge, beliefs, attitudes, and hypotheses” (Aronson, 1995; Kahneman, 2011). These biases are summarized in Table 7.3 .

Please visit this site to see a clever music video that a high school teacher made to explain these and other cognitive biases to his AP psychology students.

Were you able to determine how many marbles are needed to balance the scales in Figure 7.10 ? You need nine. Were you able to solve the problems in Figure 7.8 and Figure 7.9 ? Here are the answers ( Figure 7.11 ).

As an Amazon Associate we earn from qualifying purchases.

This book may not be used in the training of large language models or otherwise be ingested into large language models or generative AI offerings without OpenStax's permission.

Want to cite, share, or modify this book? This book uses the Creative Commons Attribution License and you must attribute OpenStax.

Access for free at https://openstax.org/books/psychology/pages/1-introduction

Authors: Rose M. Spielman, Kathryn Dumper, William Jenkins, Arlene Lacombe, Marilyn Lovett, Marion Perlmutter
Publisher/website: OpenStax
Book title: Psychology
Publication date: Dec 8, 2014
Location: Houston, Texas
Book URL: https://openstax.org/books/psychology/pages/1-introduction
Section URL: https://openstax.org/books/psychology/pages/7-3-problem-solving

© Feb 9, 2022 OpenStax. Textbook content produced by OpenStax is licensed under a Creative Commons Attribution License . The OpenStax name, OpenStax logo, OpenStax book covers, OpenStax CNX name, and OpenStax CNX logo are not subject to the Creative Commons license and may not be reproduced without the prior and express written consent of Rice University.

Ability Tests

What are ability tests.

Ability tests, also known as aptitude tests, assess various cognitive abilities such as verbal, numerical, abstract reasoning, spatial awareness, logical reasoning, and problem-solving. These tests are designed to evaluate individuals’ natural talents and potential to learn and succeed in specific areas.

For example, a verbal ability test might assess an individual’s verbal comprehension, fluency, vocabulary, and ability to understand and manipulate words. On the other hand, a numerical ability test might measure one’s numerical reasoning, mathematical skills, and ability to work with numbers accurately.

Types of Ability Tests

There are several types of ability tests, each focusing on different cognitive domains and skills. Some common types include:

Verbal Ability Tests: These assess language-related skills, including vocabulary, reading comprehension, and verbal reasoning.
Numerical Ability Tests: These evaluate an individual’s aptitude in working with numbers, mathematical problem-solving, and reasoning abilities.
Abstract Reasoning Tests: These tests measure an individual’s ability to identify patterns, think logically, and solve abstract problems.
Spatial Ability Tests: These assess one’s capacity to visualize and manipulate objects in space, such as mental rotation and spatial visualization.
Mechanical Reasoning Tests: These evaluate an individual’s understanding of mechanical concepts, principles, and problem-solving abilities.

Why Are Ability Tests Important?

Ability tests play a crucial role in various settings:

Career Selection and Recruitment: Employers use ability tests during the hiring process to identify candidates with the necessary skills and potentials for a specific job role. These tests provide objective and standardized measures to evaluate job applicants’ suitability and compatibility with the required tasks.
Educational Assessment: Educational institutions use ability tests to assess students’ aptitude for specific subjects or to identify areas where extra support may be required. These tests help educators tailor their teaching methods and curriculum to meet individual students’ needs.
Personal Development: Ability tests can assist individuals in identifying their strengths and areas for improvement. By understanding their natural abilities, individuals can make informed decisions about their career paths, personal goals, and areas to focus on for personal growth.

Key Considerations while using Ability Tests

When using ability tests, it’s important to:

Ensure Test Validity: Use tests that have been validated and are recognized by professionals in the field to ensure accurate and meaningful results.
Avoid Bias: Ensure tests are fair and unbiased, taking into account cultural and demographic differences, to avoid any form of discrimination.
Consider Multiple Factors: Combine ability test results with other assessment measures, such as interviews and work samples, for a comprehensive evaluation.
Provide Feedback and Support: Offer feedback to individuals who have taken ability tests to help them understand their results and how they can further develop their skills or seek appropriate support.

In Conclusion

Ability tests provide valuable insights into an individual’s natural talents and potential in specific areas. They are widely used in various fields to assess individuals’ cognitive abilities and inform decision-making processes. By understanding an individual’s strengths and weaknesses, ability tests contribute to effective career selections, educational assessments, and personal development. However, it’s essential to use reliable and validated tests, avoid bias, consider multiple factors, and provide appropriate support to ensure the accurate and fair assessment of an individual’s abilities.

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

7 Module 7: Thinking, Reasoning, and Problem-Solving

This module is about how a solid working knowledge of psychological principles can help you to think more effectively, so you can succeed in school and life. You might be inclined to believe that—because you have been thinking for as long as you can remember, because you are able to figure out the solution to many problems, because you feel capable of using logic to argue a point, because you can evaluate whether the things you read and hear make sense—you do not need any special training in thinking. But this, of course, is one of the key barriers to helping people think better. If you do not believe that there is anything wrong, why try to fix it?

The human brain is indeed a remarkable thinking machine, capable of amazing, complex, creative, logical thoughts. Why, then, are we telling you that you need to learn how to think? Mainly because one major lesson from cognitive psychology is that these capabilities of the human brain are relatively infrequently realized. Many psychologists believe that people are essentially “cognitive misers.” It is not that we are lazy, but that we have a tendency to expend the least amount of mental effort necessary. Although you may not realize it, it actually takes a great deal of energy to think. Careful, deliberative reasoning and critical thinking are very difficult. Because we seem to be successful without going to the trouble of using these skills well, it feels unnecessary to develop them. As you shall see, however, there are many pitfalls in the cognitive processes described in this module. When people do not devote extra effort to learning and improving reasoning, problem solving, and critical thinking skills, they make many errors.

As is true for memory, if you develop the cognitive skills presented in this module, you will be more successful in school. It is important that you realize, however, that these skills will help you far beyond school, even more so than a good memory will. Although it is somewhat useful to have a good memory, ten years from now no potential employer will care how many questions you got right on multiple choice exams during college. All of them will, however, recognize whether you are a logical, analytical, critical thinker. With these thinking skills, you will be an effective, persuasive communicator and an excellent problem solver.

The module begins by describing different kinds of thought and knowledge, especially conceptual knowledge and critical thinking. An understanding of these differences will be valuable as you progress through school and encounter different assignments that require you to tap into different kinds of knowledge. The second section covers deductive and inductive reasoning, which are processes we use to construct and evaluate strong arguments. They are essential skills to have whenever you are trying to persuade someone (including yourself) of some point, or to respond to someone’s efforts to persuade you. The module ends with a section about problem solving. A solid understanding of the key processes involved in problem solving will help you to handle many daily challenges.

7.1. Different kinds of thought

7.2. Reasoning and Judgment

7.3. Problem Solving

READING WITH PURPOSE

Remember and understand.

By reading and studying Module 7, you should be able to remember and describe:

Concepts and inferences (7.1)
Procedural knowledge (7.1)
Metacognition (7.1)
Characteristics of critical thinking: skepticism; identify biases, distortions, omissions, and assumptions; reasoning and problem solving skills (7.1)
Reasoning: deductive reasoning, deductively valid argument, inductive reasoning, inductively strong argument, availability heuristic, representativeness heuristic (7.2)
Fixation: functional fixedness, mental set (7.3)
Algorithms, heuristics, and the role of confirmation bias (7.3)
Effective problem solving sequence (7.3)

By reading and thinking about how the concepts in Module 6 apply to real life, you should be able to:

Identify which type of knowledge a piece of information is (7.1)
Recognize examples of deductive and inductive reasoning (7.2)
Recognize judgments that have probably been influenced by the availability heuristic (7.2)
Recognize examples of problem solving heuristics and algorithms (7.3)

Analyze, Evaluate, and Create

By reading and thinking about Module 6, participating in classroom activities, and completing out-of-class assignments, you should be able to:

Use the principles of critical thinking to evaluate information (7.1)
Explain whether examples of reasoning arguments are deductively valid or inductively strong (7.2)
Outline how you could try to solve a problem from your life using the effective problem solving sequence (7.3)

7.1. Different kinds of thought and knowledge

Take a few minutes to write down everything that you know about dogs.
Do you believe that:
Psychic ability exists?
Hypnosis is an altered state of consciousness?
Magnet therapy is effective for relieving pain?
Aerobic exercise is an effective treatment for depression?
UFO’s from outer space have visited earth?

On what do you base your belief or disbelief for the questions above?

Of course, we all know what is meant by the words think and knowledge . You probably also realize that they are not unitary concepts; there are different kinds of thought and knowledge. In this section, let us look at some of these differences. If you are familiar with these different kinds of thought and pay attention to them in your classes, it will help you to focus on the right goals, learn more effectively, and succeed in school. Different assignments and requirements in school call on you to use different kinds of knowledge or thought, so it will be very helpful for you to learn to recognize them (Anderson, et al. 2001).

Factual and conceptual knowledge

Module 5 introduced the idea of declarative memory, which is composed of facts and episodes. If you have ever played a trivia game or watched Jeopardy on TV, you realize that the human brain is able to hold an extraordinary number of facts. Likewise, you realize that each of us has an enormous store of episodes, essentially facts about events that happened in our own lives. It may be difficult to keep that in mind when we are struggling to retrieve one of those facts while taking an exam, however. Part of the problem is that, in contradiction to the advice from Module 5, many students continue to try to memorize course material as a series of unrelated facts (picture a history student simply trying to memorize history as a set of unrelated dates without any coherent story tying them together). Facts in the real world are not random and unorganized, however. It is the way that they are organized that constitutes a second key kind of knowledge, conceptual.

Concepts are nothing more than our mental representations of categories of things in the world. For example, think about dogs. When you do this, you might remember specific facts about dogs, such as they have fur and they bark. You may also recall dogs that you have encountered and picture them in your mind. All of this information (and more) makes up your concept of dog. You can have concepts of simple categories (e.g., triangle), complex categories (e.g., small dogs that sleep all day, eat out of the garbage, and bark at leaves), kinds of people (e.g., psychology professors), events (e.g., birthday parties), and abstract ideas (e.g., justice). Gregory Murphy (2002) refers to concepts as the “glue that holds our mental life together” (p. 1). Very simply, summarizing the world by using concepts is one of the most important cognitive tasks that we do. Our conceptual knowledge is our knowledge about the world. Individual concepts are related to each other to form a rich interconnected network of knowledge. For example, think about how the following concepts might be related to each other: dog, pet, play, Frisbee, chew toy, shoe. Or, of more obvious use to you now, how these concepts are related: working memory, long-term memory, declarative memory, procedural memory, and rehearsal? Because our minds have a natural tendency to organize information conceptually, when students try to remember course material as isolated facts, they are working against their strengths.

One last important point about concepts is that they allow you to instantly know a great deal of information about something. For example, if someone hands you a small red object and says, “here is an apple,” they do not have to tell you, “it is something you can eat.” You already know that you can eat it because it is true by virtue of the fact that the object is an apple; this is called drawing an inference , assuming that something is true on the basis of your previous knowledge (for example, of category membership or of how the world works) or logical reasoning.

Procedural knowledge

Physical skills, such as tying your shoes, doing a cartwheel, and driving a car (or doing all three at the same time, but don’t try this at home) are certainly a kind of knowledge. They are procedural knowledge, the same idea as procedural memory that you saw in Module 5. Mental skills, such as reading, debating, and planning a psychology experiment, are procedural knowledge, as well. In short, procedural knowledge is the knowledge how to do something (Cohen & Eichenbaum, 1993).

Metacognitive knowledge

Floyd used to think that he had a great memory. Now, he has a better memory. Why? Because he finally realized that his memory was not as great as he once thought it was. Because Floyd eventually learned that he often forgets where he put things, he finally developed the habit of putting things in the same place. (Unfortunately, he did not learn this lesson before losing at least 5 watches and a wedding ring.) Because he finally realized that he often forgets to do things, he finally started using the To Do list app on his phone. And so on. Floyd’s insights about the real limitations of his memory have allowed him to remember things that he used to forget.

All of us have knowledge about the way our own minds work. You may know that you have a good memory for people’s names and a poor memory for math formulas. Someone else might realize that they have difficulty remembering to do things, like stopping at the store on the way home. Others still know that they tend to overlook details. This knowledge about our own thinking is actually quite important; it is called metacognitive knowledge, or metacognition . Like other kinds of thinking skills, it is subject to error. For example, in unpublished research, one of the authors surveyed about 120 General Psychology students on the first day of the term. Among other questions, the students were asked them to predict their grade in the class and report their current Grade Point Average. Two-thirds of the students predicted that their grade in the course would be higher than their GPA. (The reality is that at our college, students tend to earn lower grades in psychology than their overall GPA.) Another example: Students routinely report that they thought they had done well on an exam, only to discover, to their dismay, that they were wrong (more on that important problem in a moment). Both errors reveal a breakdown in metacognition.

The Dunning-Kruger Effect

In general, most college students probably do not study enough. For example, using data from the National Survey of Student Engagement, Fosnacht, McCormack, and Lerma (2018) reported that first-year students at 4-year colleges in the U.S. averaged less than 14 hours per week preparing for classes. The typical suggestion is that you should spend two hours outside of class for every hour in class, or 24 – 30 hours per week for a full-time student. Clearly, students in general are nowhere near that recommended mark. Many observers, including some faculty, believe that this shortfall is a result of students being too busy or lazy. Now, it may be true that many students are too busy, with work and family obligations, for example. Others, are not particularly motivated in school, and therefore might correctly be labeled lazy. A third possible explanation, however, is that some students might not think they need to spend this much time. And this is a matter of metacognition. Consider the scenario that we mentioned above, students thinking they had done well on an exam only to discover that they did not. Justin Kruger and David Dunning examined scenarios very much like this in 1999. Kruger and Dunning gave research participants tests measuring humor, logic, and grammar. Then, they asked the participants to assess their own abilities and test performance in these areas. They found that participants in general tended to overestimate their abilities, already a problem with metacognition. Importantly, the participants who scored the lowest overestimated their abilities the most. Specifically, students who scored in the bottom quarter (averaging in the 12th percentile) thought they had scored in the 62nd percentile. This has become known as the Dunning-Kruger effect . Many individual faculty members have replicated these results with their own student on their course exams, including the authors of this book. Think about it. Some students who just took an exam and performed poorly believe that they did well before seeing their score. It seems very likely that these are the very same students who stopped studying the night before because they thought they were “done.” Quite simply, it is not just that they did not know the material. They did not know that they did not know the material. That is poor metacognition.

In order to develop good metacognitive skills, you should continually monitor your thinking and seek frequent feedback on the accuracy of your thinking (Medina, Castleberry, & Persky 2017). For example, in classes get in the habit of predicting your exam grades. As soon as possible after taking an exam, try to find out which questions you missed and try to figure out why. If you do this soon enough, you may be able to recall the way it felt when you originally answered the question. Did you feel confident that you had answered the question correctly? Then you have just discovered an opportunity to improve your metacognition. Be on the lookout for that feeling and respond with caution.

concept : a mental representation of a category of things in the world

Dunning-Kruger effect : individuals who are less competent tend to overestimate their abilities more than individuals who are more competent do

inference : an assumption about the truth of something that is not stated. Inferences come from our prior knowledge and experience, and from logical reasoning

metacognition : knowledge about one’s own cognitive processes; thinking about your thinking

Critical thinking

One particular kind of knowledge or thinking skill that is related to metacognition is critical thinking (Chew, 2020). You may have noticed that critical thinking is an objective in many college courses, and thus it could be a legitimate topic to cover in nearly any college course. It is particularly appropriate in psychology, however. As the science of (behavior and) mental processes, psychology is obviously well suited to be the discipline through which you should be introduced to this important way of thinking.

More importantly, there is a particular need to use critical thinking in psychology. We are all, in a way, experts in human behavior and mental processes, having engaged in them literally since birth. Thus, perhaps more than in any other class, students typically approach psychology with very clear ideas and opinions about its subject matter. That is, students already “know” a lot about psychology. The problem is, “it ain’t so much the things we don’t know that get us into trouble. It’s the things we know that just ain’t so” (Ward, quoted in Gilovich 1991). Indeed, many of students’ preconceptions about psychology are just plain wrong. Randolph Smith (2002) wrote a book about critical thinking in psychology called Challenging Your Preconceptions, highlighting this fact. On the other hand, many of students’ preconceptions about psychology are just plain right! But wait, how do you know which of your preconceptions are right and which are wrong? And when you come across a research finding or theory in this class that contradicts your preconceptions, what will you do? Will you stick to your original idea, discounting the information from the class? Will you immediately change your mind? Critical thinking can help us sort through this confusing mess.

But what is critical thinking? The goal of critical thinking is simple to state (but extraordinarily difficult to achieve): it is to be right, to draw the correct conclusions, to believe in things that are true and to disbelieve things that are false. We will provide two definitions of critical thinking (or, if you like, one large definition with two distinct parts). First, a more conceptual one: Critical thinking is thinking like a scientist in your everyday life (Schmaltz, Jansen, & Wenckowski, 2017). Our second definition is more operational; it is simply a list of skills that are essential to be a critical thinker. Critical thinking entails solid reasoning and problem solving skills; skepticism; and an ability to identify biases, distortions, omissions, and assumptions. Excellent deductive and inductive reasoning, and problem solving skills contribute to critical thinking. So, you can consider the subject matter of sections 7.2 and 7.3 to be part of critical thinking. Because we will be devoting considerable time to these concepts in the rest of the module, let us begin with a discussion about the other aspects of critical thinking.

Let’s address that first part of the definition. Scientists form hypotheses, or predictions about some possible future observations. Then, they collect data, or information (think of this as making those future observations). They do their best to make unbiased observations using reliable techniques that have been verified by others. Then, and only then, they draw a conclusion about what those observations mean. Oh, and do not forget the most important part. “Conclusion” is probably not the most appropriate word because this conclusion is only tentative. A scientist is always prepared that someone else might come along and produce new observations that would require a new conclusion be drawn. Wow! If you like to be right, you could do a lot worse than using a process like this.

A Critical Thinker’s Toolkit

Now for the second part of the definition. Good critical thinkers (and scientists) rely on a variety of tools to evaluate information. Perhaps the most recognizable tool for critical thinking is skepticism (and this term provides the clearest link to the thinking like a scientist definition, as you are about to see). Some people intend it as an insult when they call someone a skeptic. But if someone calls you a skeptic, if they are using the term correctly, you should consider it a great compliment. Simply put, skepticism is a way of thinking in which you refrain from drawing a conclusion or changing your mind until good evidence has been provided. People from Missouri should recognize this principle, as Missouri is known as the Show-Me State. As a skeptic, you are not inclined to believe something just because someone said so, because someone else believes it, or because it sounds reasonable. You must be persuaded by high quality evidence.

Of course, if that evidence is produced, you have a responsibility as a skeptic to change your belief. Failure to change a belief in the face of good evidence is not skepticism; skepticism has open mindedness at its core. M. Neil Browne and Stuart Keeley (2018) use the term weak sense critical thinking to describe critical thinking behaviors that are used only to strengthen a prior belief. Strong sense critical thinking, on the other hand, has as its goal reaching the best conclusion. Sometimes that means strengthening your prior belief, but sometimes it means changing your belief to accommodate the better evidence.

Many times, a failure to think critically or weak sense critical thinking is related to a bias , an inclination, tendency, leaning, or prejudice. Everybody has biases, but many people are unaware of them. Awareness of your own biases gives you the opportunity to control or counteract them. Unfortunately, however, many people are happy to let their biases creep into their attempts to persuade others; indeed, it is a key part of their persuasive strategy. To see how these biases influence messages, just look at the different descriptions and explanations of the same events given by people of different ages or income brackets, or conservative versus liberal commentators, or by commentators from different parts of the world. Of course, to be successful, these people who are consciously using their biases must disguise them. Even undisguised biases can be difficult to identify, so disguised ones can be nearly impossible.

Here are some common sources of biases:

Personal values and beliefs. Some people believe that human beings are basically driven to seek power and that they are typically in competition with one another over scarce resources. These beliefs are similar to the world-view that political scientists call “realism.” Other people believe that human beings prefer to cooperate and that, given the chance, they will do so. These beliefs are similar to the world-view known as “idealism.” For many people, these deeply held beliefs can influence, or bias, their interpretations of such wide ranging situations as the behavior of nations and their leaders or the behavior of the driver in the car ahead of you. For example, if your worldview is that people are typically in competition and someone cuts you off on the highway, you may assume that the driver did it purposely to get ahead of you. Other types of beliefs about the way the world is or the way the world should be, for example, political beliefs, can similarly become a significant source of bias.
Racism, sexism, ageism and other forms of prejudice and bigotry. These are, sadly, a common source of bias in many people. They are essentially a special kind of “belief about the way the world is.” These beliefs—for example, that women do not make effective leaders—lead people to ignore contradictory evidence (examples of effective women leaders, or research that disputes the belief) and to interpret ambiguous evidence in a way consistent with the belief.
Self-interest. When particular people benefit from things turning out a certain way, they can sometimes be very susceptible to letting that interest bias them. For example, a company that will earn a profit if they sell their product may have a bias in the way that they give information about their product. A union that will benefit if its members get a generous contract might have a bias in the way it presents information about salaries at competing organizations. (Note that our inclusion of examples describing both companies and unions is an explicit attempt to control for our own personal biases). Home buyers are often dismayed to discover that they purchased their dream house from someone whose self-interest led them to lie about flooding problems in the basement or back yard. This principle, the biasing power of self-interest, is likely what led to the famous phrase Caveat Emptor (let the buyer beware) .

Knowing that these types of biases exist will help you evaluate evidence more critically. Do not forget, though, that people are not always keen to let you discover the sources of biases in their arguments. For example, companies or political organizations can sometimes disguise their support of a research study by contracting with a university professor, who comes complete with a seemingly unbiased institutional affiliation, to conduct the study.

People’s biases, conscious or unconscious, can lead them to make omissions, distortions, and assumptions that undermine our ability to correctly evaluate evidence. It is essential that you look for these elements. Always ask, what is missing, what is not as it appears, and what is being assumed here? For example, consider this (fictional) chart from an ad reporting customer satisfaction at 4 local health clubs.

Clearly, from the results of the chart, one would be tempted to give Club C a try, as customer satisfaction is much higher than for the other 3 clubs.

There are so many distortions and omissions in this chart, however, that it is actually quite meaningless. First, how was satisfaction measured? Do the bars represent responses to a survey? If so, how were the questions asked? Most importantly, where is the missing scale for the chart? Although the differences look quite large, are they really?

Well, here is the same chart, with a different scale, this time labeled:

Club C is not so impressive any more, is it? In fact, all of the health clubs have customer satisfaction ratings (whatever that means) between 85% and 88%. In the first chart, the entire scale of the graph included only the percentages between 83 and 89. This “judicious” choice of scale—some would call it a distortion—and omission of that scale from the chart make the tiny differences among the clubs seem important, however.

Also, in order to be a critical thinker, you need to learn to pay attention to the assumptions that underlie a message. Let us briefly illustrate the role of assumptions by touching on some people’s beliefs about the criminal justice system in the US. Some believe that a major problem with our judicial system is that many criminals go free because of legal technicalities. Others believe that a major problem is that many innocent people are convicted of crimes. The simple fact is, both types of errors occur. A person’s conclusion about which flaw in our judicial system is the greater tragedy is based on an assumption about which of these is the more serious error (letting the guilty go free or convicting the innocent). This type of assumption is called a value assumption (Browne and Keeley, 2018). It reflects the differences in values that people develop, differences that may lead us to disregard valid evidence that does not fit in with our particular values.

Oh, by the way, some students probably noticed this, but the seven tips for evaluating information that we shared in Module 1 are related to this. Actually, they are part of this section. The tips are, to a very large degree, set of ideas you can use to help you identify biases, distortions, omissions, and assumptions. If you do not remember this section, we strongly recommend you take a few minutes to review it.

skepticism : a way of thinking in which you refrain from drawing a conclusion or changing your mind until good evidence has been provided

bias : an inclination, tendency, leaning, or prejudice

Which of your beliefs (or disbeliefs) from the Activate exercise for this section were derived from a process of critical thinking? If some of your beliefs were not based on critical thinking, are you willing to reassess these beliefs? If the answer is no, why do you think that is? If the answer is yes, what concrete steps will you take?

7.2 Reasoning and Judgment

What percentage of kidnappings are committed by strangers?
Which area of the house is riskiest: kitchen, bathroom, or stairs?
What is the most common cancer in the US?
What percentage of workplace homicides are committed by co-workers?

An essential set of procedural thinking skills is reasoning , the ability to generate and evaluate solid conclusions from a set of statements or evidence. You should note that these conclusions (when they are generated instead of being evaluated) are one key type of inference that we described in Section 7.1. There are two main types of reasoning, deductive and inductive.

Deductive reasoning

Suppose your teacher tells you that if you get an A on the final exam in a course, you will get an A for the whole course. Then, you get an A on the final exam. What will your final course grade be? Most people can see instantly that you can conclude with certainty that you will get an A for the course. This is a type of reasoning called deductive reasoning , which is defined as reasoning in which a conclusion is guaranteed to be true as long as the statements leading to it are true. The three statements can be listed as an argument , with two beginning statements and a conclusion:

Statement 1: If you get an A on the final exam, you will get an A for the course

Statement 2: You get an A on the final exam

Conclusion: You will get an A for the course

This particular arrangement, in which true beginning statements lead to a guaranteed true conclusion, is known as a deductively valid argument . Although deductive reasoning is often the subject of abstract, brain-teasing, puzzle-like word problems, it is actually an extremely important type of everyday reasoning. It is just hard to recognize sometimes. For example, imagine that you are looking for your car keys and you realize that they are either in the kitchen drawer or in your book bag. After looking in the kitchen drawer, you instantly know that they must be in your book bag. That conclusion results from a simple deductive reasoning argument. In addition, solid deductive reasoning skills are necessary for you to succeed in the sciences, philosophy, math, computer programming, and any endeavor involving the use of logic to persuade others to your point of view or to evaluate others’ arguments.

Cognitive psychologists, and before them philosophers, have been quite interested in deductive reasoning, not so much for its practical applications, but for the insights it can offer them about the ways that human beings think. One of the early ideas to emerge from the examination of deductive reasoning is that people learn (or develop) mental versions of rules that allow them to solve these types of reasoning problems (Braine, 1978; Braine, Reiser, & Rumain, 1984). The best way to see this point of view is to realize that there are different possible rules, and some of them are very simple. For example, consider this rule of logic:

therefore q

Logical rules are often presented abstractly, as letters, in order to imply that they can be used in very many specific situations. Here is a concrete version of the of the same rule:

I’ll either have pizza or a hamburger for dinner tonight (p or q)

I won’t have pizza (not p)

Therefore, I’ll have a hamburger (therefore q)

This kind of reasoning seems so natural, so easy, that it is quite plausible that we would use a version of this rule in our daily lives. At least, it seems more plausible than some of the alternative possibilities—for example, that we need to have experience with the specific situation (pizza or hamburger, in this case) in order to solve this type of problem easily. So perhaps there is a form of natural logic (Rips, 1990) that contains very simple versions of logical rules. When we are faced with a reasoning problem that maps onto one of these rules, we use the rule.

But be very careful; things are not always as easy as they seem. Even these simple rules are not so simple. For example, consider the following rule. Many people fail to realize that this rule is just as valid as the pizza or hamburger rule above.

if p, then q

therefore, not p

Concrete version:

If I eat dinner, then I will have dessert

I did not have dessert

Therefore, I did not eat dinner

The simple fact is, it can be very difficult for people to apply rules of deductive logic correctly; as a result, they make many errors when trying to do so. Is this a deductively valid argument or not?

Students who like school study a lot

Students who study a lot get good grades

Jane does not like school

Therefore, Jane does not get good grades

Many people are surprised to discover that this is not a logically valid argument; the conclusion is not guaranteed to be true from the beginning statements. Although the first statement says that students who like school study a lot, it does NOT say that students who do not like school do not study a lot. In other words, it may very well be possible to study a lot without liking school. Even people who sometimes get problems like this right might not be using the rules of deductive reasoning. Instead, they might just be making judgments for examples they know, in this case, remembering instances of people who get good grades despite not liking school.

Making deductive reasoning even more difficult is the fact that there are two important properties that an argument may have. One, it can be valid or invalid (meaning that the conclusion does or does not follow logically from the statements leading up to it). Two, an argument (or more correctly, its conclusion) can be true or false. Here is an example of an argument that is logically valid, but has a false conclusion (at least we think it is false).

Either you are eleven feet tall or the Grand Canyon was created by a spaceship crashing into the earth.

You are not eleven feet tall

Therefore the Grand Canyon was created by a spaceship crashing into the earth

This argument has the exact same form as the pizza or hamburger argument above, making it is deductively valid. The conclusion is so false, however, that it is absurd (of course, the reason the conclusion is false is that the first statement is false). When people are judging arguments, they tend to not observe the difference between deductive validity and the empirical truth of statements or conclusions. If the elements of an argument happen to be true, people are likely to judge the argument logically valid; if the elements are false, they will very likely judge it invalid (Markovits & Bouffard-Bouchard, 1992; Moshman & Franks, 1986). Thus, it seems a stretch to say that people are using these logical rules to judge the validity of arguments. Many psychologists believe that most people actually have very limited deductive reasoning skills (Johnson-Laird, 1999). They argue that when faced with a problem for which deductive logic is required, people resort to some simpler technique, such as matching terms that appear in the statements and the conclusion (Evans, 1982). This might not seem like a problem, but what if reasoners believe that the elements are true and they happen to be wrong; they will would believe that they are using a form of reasoning that guarantees they are correct and yet be wrong.

deductive reasoning : a type of reasoning in which the conclusion is guaranteed to be true any time the statements leading up to it are true

argument : a set of statements in which the beginning statements lead to a conclusion

deductively valid argument : an argument for which true beginning statements guarantee that the conclusion is true

Inductive reasoning and judgment

Every day, you make many judgments about the likelihood of one thing or another. Whether you realize it or not, you are practicing inductive reasoning on a daily basis. In inductive reasoning arguments, a conclusion is likely whenever the statements preceding it are true. The first thing to notice about inductive reasoning is that, by definition, you can never be sure about your conclusion; you can only estimate how likely the conclusion is. Inductive reasoning may lead you to focus on Memory Encoding and Recoding when you study for the exam, but it is possible the instructor will ask more questions about Memory Retrieval instead. Unlike deductive reasoning, the conclusions you reach through inductive reasoning are only probable, not certain. That is why scientists consider inductive reasoning weaker than deductive reasoning. But imagine how hard it would be for us to function if we could not act unless we were certain about the outcome.

Inductive reasoning can be represented as logical arguments consisting of statements and a conclusion, just as deductive reasoning can be. In an inductive argument, you are given some statements and a conclusion (or you are given some statements and must draw a conclusion). An argument is inductively strong if the conclusion would be very probable whenever the statements are true. So, for example, here is an inductively strong argument:

Statement #1: The forecaster on Channel 2 said it is going to rain today.
Statement #2: The forecaster on Channel 5 said it is going to rain today.
Statement #3: It is very cloudy and humid.
Statement #4: You just heard thunder.
Conclusion (or judgment): It is going to rain today.

Think of the statements as evidence, on the basis of which you will draw a conclusion. So, based on the evidence presented in the four statements, it is very likely that it will rain today. Will it definitely rain today? Certainly not. We can all think of times that the weather forecaster was wrong.

A true story: Some years ago psychology student was watching a baseball playoff game between the St. Louis Cardinals and the Los Angeles Dodgers. A graphic on the screen had just informed the audience that the Cardinal at bat, (Hall of Fame shortstop) Ozzie Smith, a switch hitter batting left-handed for this plate appearance, had never, in nearly 3000 career at-bats, hit a home run left-handed. The student, who had just learned about inductive reasoning in his psychology class, turned to his companion (a Cardinals fan) and smugly said, “It is an inductively strong argument that Ozzie Smith will not hit a home run.” He turned back to face the television just in time to watch the ball sail over the right field fence for a home run. Although the student felt foolish at the time, he was not wrong. It was an inductively strong argument; 3000 at-bats is an awful lot of evidence suggesting that the Wizard of Ozz (as he was known) would not be hitting one out of the park (think of each at-bat without a home run as a statement in an inductive argument). Sadly (for the die-hard Cubs fan and Cardinals-hating student), despite the strength of the argument, the conclusion was wrong.

Given the possibility that we might draw an incorrect conclusion even with an inductively strong argument, we really want to be sure that we do, in fact, make inductively strong arguments. If we judge something probable, it had better be probable. If we judge something nearly impossible, it had better not happen. Think of inductive reasoning, then, as making reasonably accurate judgments of the probability of some conclusion given a set of evidence.

We base many decisions in our lives on inductive reasoning. For example:

Statement #1: Psychology is not my best subject

Statement #2: My psychology instructor has a reputation for giving difficult exams

Statement #3: My first psychology exam was much harder than I expected

Judgment: The next exam will probably be very difficult.

Decision: I will study tonight instead of watching Netflix.

Some other examples of judgments that people commonly make in a school context include judgments of the likelihood that:

A particular class will be interesting/useful/difficult
You will be able to finish writing a paper by next week if you go out tonight
Your laptop’s battery will last through the next trip to the library
You will not miss anything important if you skip class tomorrow
Your instructor will not notice if you skip class tomorrow
You will be able to find a book that you will need for a paper
There will be an essay question about Memory Encoding on the next exam

Tversky and Kahneman (1983) recognized that there are two general ways that we might make these judgments; they termed them extensional (i.e., following the laws of probability) and intuitive (i.e., using shortcuts or heuristics, see below). We will use a similar distinction between Type 1 and Type 2 thinking, as described by Keith Stanovich and his colleagues (Evans and Stanovich, 2013; Stanovich and West, 2000). Type 1 thinking is fast, automatic, effortful, and emotional. In fact, it is hardly fair to call it reasoning at all, as judgments just seem to pop into one’s head. Type 2 thinking , on the other hand, is slow, effortful, and logical. So obviously, it is more likely to lead to a correct judgment, or an optimal decision. The problem is, we tend to over-rely on Type 1. Now, we are not saying that Type 2 is the right way to go for every decision or judgment we make. It seems a bit much, for example, to engage in a step-by-step logical reasoning procedure to decide whether we will have chicken or fish for dinner tonight.

Many bad decisions in some very important contexts, however, can be traced back to poor judgments of the likelihood of certain risks or outcomes that result from the use of Type 1 when a more logical reasoning process would have been more appropriate. For example:

Statement #1: It is late at night.

Statement #2: Albert has been drinking beer for the past five hours at a party.

Statement #3: Albert is not exactly sure where he is or how far away home is.

Judgment: Albert will have no difficulty walking home.

Decision: He walks home alone.

As you can see in this example, the three statements backing up the judgment do not really support it. In other words, this argument is not inductively strong because it is based on judgments that ignore the laws of probability. What are the chances that someone facing these conditions will be able to walk home alone easily? And one need not be drunk to make poor decisions based on judgments that just pop into our heads.

The truth is that many of our probability judgments do not come very close to what the laws of probability say they should be. Think about it. In order for us to reason in accordance with these laws, we would need to know the laws of probability, which would allow us to calculate the relationship between particular pieces of evidence and the probability of some outcome (i.e., how much likelihood should change given a piece of evidence), and we would have to do these heavy math calculations in our heads. After all, that is what Type 2 requires. Needless to say, even if we were motivated, we often do not even know how to apply Type 2 reasoning in many cases.

So what do we do when we don’t have the knowledge, skills, or time required to make the correct mathematical judgment? Do we hold off and wait until we can get better evidence? Do we read up on probability and fire up our calculator app so we can compute the correct probability? Of course not. We rely on Type 1 thinking. We “wing it.” That is, we come up with a likelihood estimate using some means at our disposal. Psychologists use the term heuristic to describe the type of “winging it” we are talking about. A heuristic is a shortcut strategy that we use to make some judgment or solve some problem (see Section 7.3). Heuristics are easy and quick, think of them as the basic procedures that are characteristic of Type 1. They can absolutely lead to reasonably good judgments and decisions in some situations (like choosing between chicken and fish for dinner). They are, however, far from foolproof. There are, in fact, quite a lot of situations in which heuristics can lead us to make incorrect judgments, and in many cases the decisions based on those judgments can have serious consequences.

Let us return to the activity that begins this section. You were asked to judge the likelihood (or frequency) of certain events and risks. You were free to come up with your own evidence (or statements) to make these judgments. This is where a heuristic crops up. As a judgment shortcut, we tend to generate specific examples of those very events to help us decide their likelihood or frequency. For example, if we are asked to judge how common, frequent, or likely a particular type of cancer is, many of our statements would be examples of specific cancer cases:

Statement #1: Andy Kaufman (comedian) had lung cancer.

Statement #2: Colin Powell (US Secretary of State) had prostate cancer.

Statement #3: Bob Marley (musician) had skin and brain cancer

Statement #4: Sandra Day O’Connor (Supreme Court Justice) had breast cancer.

Statement #5: Fred Rogers (children’s entertainer) had stomach cancer.

Statement #6: Robin Roberts (news anchor) had breast cancer.

Statement #7: Bette Davis (actress) had breast cancer.

Judgment: Breast cancer is the most common type.

Your own experience or memory may also tell you that breast cancer is the most common type. But it is not (although it is common). Actually, skin cancer is the most common type in the US. We make the same types of misjudgments all the time because we do not generate the examples or evidence according to their actual frequencies or probabilities. Instead, we have a tendency (or bias) to search for the examples in memory; if they are easy to retrieve, we assume that they are common. To rephrase this in the language of the heuristic, events seem more likely to the extent that they are available to memory. This bias has been termed the availability heuristic (Kahneman and Tversky, 1974).

The fact that we use the availability heuristic does not automatically mean that our judgment is wrong. The reason we use heuristics in the first place is that they work fairly well in many cases (and, of course that they are easy to use). So, the easiest examples to think of sometimes are the most common ones. Is it more likely that a member of the U.S. Senate is a man or a woman? Most people have a much easier time generating examples of male senators. And as it turns out, the U.S. Senate has many more men than women (74 to 26 in 2020). In this case, then, the availability heuristic would lead you to make the correct judgment; it is far more likely that a senator would be a man.

In many other cases, however, the availability heuristic will lead us astray. This is because events can be memorable for many reasons other than their frequency. Section 5.2, Encoding Meaning, suggested that one good way to encode the meaning of some information is to form a mental image of it. Thus, information that has been pictured mentally will be more available to memory. Indeed, an event that is vivid and easily pictured will trick many people into supposing that type of event is more common than it actually is. Repetition of information will also make it more memorable. So, if the same event is described to you in a magazine, on the evening news, on a podcast that you listen to, and in your Facebook feed; it will be very available to memory. Again, the availability heuristic will cause you to misperceive the frequency of these types of events.

Most interestingly, information that is unusual is more memorable. Suppose we give you the following list of words to remember: box, flower, letter, platypus, oven, boat, newspaper, purse, drum, car. Very likely, the easiest word to remember would be platypus, the unusual one. The same thing occurs with memories of events. An event may be available to memory because it is unusual, yet the availability heuristic leads us to judge that the event is common. Did you catch that? In these cases, the availability heuristic makes us think the exact opposite of the true frequency. We end up thinking something is common because it is unusual (and therefore memorable). Yikes.

The misapplication of the availability heuristic sometimes has unfortunate results. For example, if you went to K-12 school in the US over the past 10 years, it is extremely likely that you have participated in lockdown and active shooter drills. Of course, everyone is trying to prevent the tragedy of another school shooting. And believe us, we are not trying to minimize how terrible the tragedy is. But the truth of the matter is, school shootings are extremely rare. Because the federal government does not keep a database of school shootings, the Washington Post has maintained their own running tally. Between 1999 and January 2020 (the date of the most recent school shooting with a death in the US at of the time this paragraph was written), the Post reported a total of 254 people died in school shootings in the US. Not 254 per year, 254 total. That is an average of 12 per year. Of course, that is 254 people who should not have died (particularly because many were children), but in a country with approximately 60,000,000 students and teachers, this is a very small risk.

But many students and teachers are terrified that they will be victims of school shootings because of the availability heuristic. It is so easy to think of examples (they are very available to memory) that people believe the event is very common. It is not. And there is a downside to this. We happen to believe that there is an enormous gun violence problem in the United States. According the the Centers for Disease Control and Prevention, there were 39,773 firearm deaths in the US in 2017. Fifteen of those deaths were in school shootings, according to the Post. 60% of those deaths were suicides. When people pay attention to the school shooting risk (low), they often fail to notice the much larger risk.

And examples like this are by no means unique. The authors of this book have been teaching psychology since the 1990’s. We have been able to make the exact same arguments about the misapplication of the availability heuristics and keep them current by simply swapping out for the “fear of the day.” In the 1990’s it was children being kidnapped by strangers (it was known as “stranger danger”) despite the facts that kidnappings accounted for only 2% of the violent crimes committed against children, and only 24% of kidnappings are committed by strangers (US Department of Justice, 2007). This fear overlapped with the fear of terrorism that gripped the country after the 2001 terrorist attacks on the World Trade Center and US Pentagon and still plagues the population of the US somewhat in 2020. After a well-publicized, sensational act of violence, people are extremely likely to increase their estimates of the chances that they, too, will be victims of terror. Think about the reality, however. In October of 2001, a terrorist mailed anthrax spores to members of the US government and a number of media companies. A total of five people died as a result of this attack. The nation was nearly paralyzed by the fear of dying from the attack; in reality the probability of an individual person dying was 0.00000002.

The availability heuristic can lead you to make incorrect judgments in a school setting as well. For example, suppose you are trying to decide if you should take a class from a particular math professor. You might try to make a judgment of how good a teacher she is by recalling instances of friends and acquaintances making comments about her teaching skill. You may have some examples that suggest that she is a poor teacher very available to memory, so on the basis of the availability heuristic you judge her a poor teacher and decide to take the class from someone else. What if, however, the instances you recalled were all from the same person, and this person happens to be a very colorful storyteller? The subsequent ease of remembering the instances might not indicate that the professor is a poor teacher after all.

Although the availability heuristic is obviously important, it is not the only judgment heuristic we use. Amos Tversky and Daniel Kahneman examined the role of heuristics in inductive reasoning in a long series of studies. Kahneman received a Nobel Prize in Economics for this research in 2002, and Tversky would have certainly received one as well if he had not died of melanoma at age 59 in 1996 (Nobel Prizes are not awarded posthumously). Kahneman and Tversky demonstrated repeatedly that people do not reason in ways that are consistent with the laws of probability. They identified several heuristic strategies that people use instead to make judgments about likelihood. The importance of this work for economics (and the reason that Kahneman was awarded the Nobel Prize) is that earlier economic theories had assumed that people do make judgments rationally, that is, in agreement with the laws of probability.

Another common heuristic that people use for making judgments is the representativeness heuristic (Kahneman & Tversky 1973). Suppose we describe a person to you. He is quiet and shy, has an unassuming personality, and likes to work with numbers. Is this person more likely to be an accountant or an attorney? If you said accountant, you were probably using the representativeness heuristic. Our imaginary person is judged likely to be an accountant because he resembles, or is representative of the concept of, an accountant. When research participants are asked to make judgments such as these, the only thing that seems to matter is the representativeness of the description. For example, if told that the person described is in a room that contains 70 attorneys and 30 accountants, participants will still assume that he is an accountant.

inductive reasoning : a type of reasoning in which we make judgments about likelihood from sets of evidence

inductively strong argument : an inductive argument in which the beginning statements lead to a conclusion that is probably true

heuristic : a shortcut strategy that we use to make judgments and solve problems. Although they are easy to use, they do not guarantee correct judgments and solutions

availability heuristic : judging the frequency or likelihood of some event type according to how easily examples of the event can be called to mind (i.e., how available they are to memory)

representativeness heuristic: judging the likelihood that something is a member of a category on the basis of how much it resembles a typical category member (i.e., how representative it is of the category)

Type 1 thinking : fast, automatic, and emotional thinking.

Type 2 thinking : slow, effortful, and logical thinking.

What percentage of workplace homicides are co-worker violence?

Many people get these questions wrong. The answers are 10%; stairs; skin; 6%. How close were your answers? Explain how the availability heuristic might have led you to make the incorrect judgments.

Can you think of some other judgments that you have made (or beliefs that you have) that might have been influenced by the availability heuristic?

7.3 Problem Solving

Please take a few minutes to list a number of problems that you are facing right now.
Now write about a problem that you recently solved.
What is your definition of a problem?

Mary has a problem. Her daughter, ordinarily quite eager to please, appears to delight in being the last person to do anything. Whether getting ready for school, going to piano lessons or karate class, or even going out with her friends, she seems unwilling or unable to get ready on time. Other people have different kinds of problems. For example, many students work at jobs, have numerous family commitments, and are facing a course schedule full of difficult exams, assignments, papers, and speeches. How can they find enough time to devote to their studies and still fulfill their other obligations? Speaking of students and their problems: Show that a ball thrown vertically upward with initial velocity v0 takes twice as much time to return as to reach the highest point (from Spiegel, 1981).

These are three very different situations, but we have called them all problems. What makes them all the same, despite the differences? A psychologist might define a problem as a situation with an initial state, a goal state, and a set of possible intermediate states. Somewhat more meaningfully, we might consider a problem a situation in which you are in here one state (e.g., daughter is always late), you want to be there in another state (e.g., daughter is not always late), and with no obvious way to get from here to there. Defined this way, each of the three situations we outlined can now be seen as an example of the same general concept, a problem. At this point, you might begin to wonder what is not a problem, given such a general definition. It seems that nearly every non-routine task we engage in could qualify as a problem. As long as you realize that problems are not necessarily bad (it can be quite fun and satisfying to rise to the challenge and solve a problem), this may be a useful way to think about it.

Can we identify a set of problem-solving skills that would apply to these very different kinds of situations? That task, in a nutshell, is a major goal of this section. Let us try to begin to make sense of the wide variety of ways that problems can be solved with an important observation: the process of solving problems can be divided into two key parts. First, people have to notice, comprehend, and represent the problem properly in their minds (called problem representation ). Second, they have to apply some kind of solution strategy to the problem. Psychologists have studied both of these key parts of the process in detail.

When you first think about the problem-solving process, you might guess that most of our difficulties would occur because we are failing in the second step, the application of strategies. Although this can be a significant difficulty much of the time, the more important source of difficulty is probably problem representation. In short, we often fail to solve a problem because we are looking at it, or thinking about it, the wrong way.

problem : a situation in which we are in an initial state, have a desired goal state, and there is a number of possible intermediate states (i.e., there is no obvious way to get from the initial to the goal state)

problem representation : noticing, comprehending and forming a mental conception of a problem

Defining and Mentally Representing Problems in Order to Solve Them

So, the main obstacle to solving a problem is that we do not clearly understand exactly what the problem is. Recall the problem with Mary’s daughter always being late. One way to represent, or to think about, this problem is that she is being defiant. She refuses to get ready in time. This type of representation or definition suggests a particular type of solution. Another way to think about the problem, however, is to consider the possibility that she is simply being sidetracked by interesting diversions. This different conception of what the problem is (i.e., different representation) suggests a very different solution strategy. For example, if Mary defines the problem as defiance, she may be tempted to solve the problem using some kind of coercive tactics, that is, to assert her authority as her mother and force her to listen. On the other hand, if Mary defines the problem as distraction, she may try to solve it by simply removing the distracting objects.

As you might guess, when a problem is represented one way, the solution may seem very difficult, or even impossible. Seen another way, the solution might be very easy. For example, consider the following problem (from Nasar, 1998):

Two bicyclists start 20 miles apart and head toward each other, each going at a steady rate of 10 miles per hour. At the same time, a fly that travels at a steady 15 miles per hour starts from the front wheel of the southbound bicycle and flies to the front wheel of the northbound one, then turns around and flies to the front wheel of the southbound one again, and continues in this manner until he is crushed between the two front wheels. Question: what total distance did the fly cover?

Please take a few minutes to try to solve this problem.

Most people represent this problem as a question about a fly because, well, that is how the question is asked. The solution, using this representation, is to figure out how far the fly travels on the first leg of its journey, then add this total to how far it travels on the second leg of its journey (when it turns around and returns to the first bicycle), then continue to add the smaller distance from each leg of the journey until you converge on the correct answer. You would have to be quite skilled at math to solve this problem, and you would probably need some time and pencil and paper to do it.

If you consider a different representation, however, you can solve this problem in your head. Instead of thinking about it as a question about a fly, think about it as a question about the bicycles. They are 20 miles apart, and each is traveling 10 miles per hour. How long will it take for the bicycles to reach each other? Right, one hour. The fly is traveling 15 miles per hour; therefore, it will travel a total of 15 miles back and forth in the hour before the bicycles meet. Represented one way (as a problem about a fly), the problem is quite difficult. Represented another way (as a problem about two bicycles), it is easy. Changing your representation of a problem is sometimes the best—sometimes the only—way to solve it.

Unfortunately, however, changing a problem’s representation is not the easiest thing in the world to do. Often, problem solvers get stuck looking at a problem one way. This is called fixation . Most people who represent the preceding problem as a problem about a fly probably do not pause to reconsider, and consequently change, their representation. A parent who thinks her daughter is being defiant is unlikely to consider the possibility that her behavior is far less purposeful.

Problem-solving fixation was examined by a group of German psychologists called Gestalt psychologists during the 1930’s and 1940’s. Karl Dunker, for example, discovered an important type of failure to take a different perspective called functional fixedness . Imagine being a participant in one of his experiments. You are asked to figure out how to mount two candles on a door and are given an assortment of odds and ends, including a small empty cardboard box and some thumbtacks. Perhaps you have already figured out a solution: tack the box to the door so it forms a platform, then put the candles on top of the box. Most people are able to arrive at this solution. Imagine a slight variation of the procedure, however. What if, instead of being empty, the box had matches in it? Most people given this version of the problem do not arrive at the solution given above. Why? Because it seems to people that when the box contains matches, it already has a function; it is a matchbox. People are unlikely to consider a new function for an object that already has a function. This is functional fixedness.

Mental set is a type of fixation in which the problem solver gets stuck using the same solution strategy that has been successful in the past, even though the solution may no longer be useful. It is commonly seen when students do math problems for homework. Often, several problems in a row require the reapplication of the same solution strategy. Then, without warning, the next problem in the set requires a new strategy. Many students attempt to apply the formerly successful strategy on the new problem and therefore cannot come up with a correct answer.

The thing to remember is that you cannot solve a problem unless you correctly identify what it is to begin with (initial state) and what you want the end result to be (goal state). That may mean looking at the problem from a different angle and representing it in a new way. The correct representation does not guarantee a successful solution, but it certainly puts you on the right track.

A bit more optimistically, the Gestalt psychologists discovered what may be considered the opposite of fixation, namely insight . Sometimes the solution to a problem just seems to pop into your head. Wolfgang Kohler examined insight by posing many different problems to chimpanzees, principally problems pertaining to their acquisition of out-of-reach food. In one version, a banana was placed outside of a chimpanzee’s cage and a short stick inside the cage. The stick was too short to retrieve the banana, but was long enough to retrieve a longer stick also located outside of the cage. This second stick was long enough to retrieve the banana. After trying, and failing, to reach the banana with the shorter stick, the chimpanzee would try a couple of random-seeming attempts, react with some apparent frustration or anger, then suddenly rush to the longer stick, the correct solution fully realized at this point. This sudden appearance of the solution, observed many times with many different problems, was termed insight by Kohler.

Lest you think it pertains to chimpanzees only, Karl Dunker demonstrated that children also solve problems through insight in the 1930s. More importantly, you have probably experienced insight yourself. Think back to a time when you were trying to solve a difficult problem. After struggling for a while, you gave up. Hours later, the solution just popped into your head, perhaps when you were taking a walk, eating dinner, or lying in bed.

fixation : when a problem solver gets stuck looking at a problem a particular way and cannot change his or her representation of it (or his or her intended solution strategy)

functional fixedness : a specific type of fixation in which a problem solver cannot think of a new use for an object that already has a function

mental set : a specific type of fixation in which a problem solver gets stuck using the same solution strategy that has been successful in the past

insight : a sudden realization of a solution to a problem

Solving Problems by Trial and Error

Correctly identifying the problem and your goal for a solution is a good start, but recall the psychologist’s definition of a problem: it includes a set of possible intermediate states. Viewed this way, a problem can be solved satisfactorily only if one can find a path through some of these intermediate states to the goal. Imagine a fairly routine problem, finding a new route to school when your ordinary route is blocked (by road construction, for example). At each intersection, you may turn left, turn right, or go straight. A satisfactory solution to the problem (of getting to school) is a sequence of selections at each intersection that allows you to wind up at school.

If you had all the time in the world to get to school, you might try choosing intermediate states randomly. At one corner you turn left, the next you go straight, then you go left again, then right, then right, then straight. Unfortunately, trial and error will not necessarily get you where you want to go, and even if it does, it is not the fastest way to get there. For example, when a friend of ours was in college, he got lost on the way to a concert and attempted to find the venue by choosing streets to turn onto randomly (this was long before the use of GPS). Amazingly enough, the strategy worked, although he did end up missing two out of the three bands who played that night.

Trial and error is not all bad, however. B.F. Skinner, a prominent behaviorist psychologist, suggested that people often behave randomly in order to see what effect the behavior has on the environment and what subsequent effect this environmental change has on them. This seems particularly true for the very young person. Picture a child filling a household’s fish tank with toilet paper, for example. To a child trying to develop a repertoire of creative problem-solving strategies, an odd and random behavior might be just the ticket. Eventually, the exasperated parent hopes, the child will discover that many of these random behaviors do not successfully solve problems; in fact, in many cases they create problems. Thus, one would expect a decrease in this random behavior as a child matures. You should realize, however, that the opposite extreme is equally counterproductive. If the children become too rigid, never trying something unexpected and new, their problem solving skills can become too limited.

Effective problem solving seems to call for a happy medium that strikes a balance between using well-founded old strategies and trying new ground and territory. The individual who recognizes a situation in which an old problem-solving strategy would work best, and who can also recognize a situation in which a new untested strategy is necessary is halfway to success.

Solving Problems with Algorithms and Heuristics

For many problems there is a possible strategy available that will guarantee a correct solution. For example, think about math problems. Math lessons often consist of step-by-step procedures that can be used to solve the problems. If you apply the strategy without error, you are guaranteed to arrive at the correct solution to the problem. This approach is called using an algorithm , a term that denotes the step-by-step procedure that guarantees a correct solution. Because algorithms are sometimes available and come with a guarantee, you might think that most people use them frequently. Unfortunately, however, they do not. As the experience of many students who have struggled through math classes can attest, algorithms can be extremely difficult to use, even when the problem solver knows which algorithm is supposed to work in solving the problem. In problems outside of math class, we often do not even know if an algorithm is available. It is probably fair to say, then, that algorithms are rarely used when people try to solve problems.

Because algorithms are so difficult to use, people often pass up the opportunity to guarantee a correct solution in favor of a strategy that is much easier to use and yields a reasonable chance of coming up with a correct solution. These strategies are called problem solving heuristics . Similar to what you saw in section 6.2 with reasoning heuristics, a problem solving heuristic is a shortcut strategy that people use when trying to solve problems. It usually works pretty well, but does not guarantee a correct solution to the problem. For example, one problem solving heuristic might be “always move toward the goal” (so when trying to get to school when your regular route is blocked, you would always turn in the direction you think the school is). A heuristic that people might use when doing math homework is “use the same solution strategy that you just used for the previous problem.”

By the way, we hope these last two paragraphs feel familiar to you. They seem to parallel a distinction that you recently learned. Indeed, algorithms and problem-solving heuristics are another example of the distinction between Type 1 thinking and Type 2 thinking.

Although it is probably not worth describing a large number of specific heuristics, two observations about heuristics are worth mentioning. First, heuristics can be very general or they can be very specific, pertaining to a particular type of problem only. For example, “always move toward the goal” is a general strategy that you can apply to countless problem situations. On the other hand, “when you are lost without a functioning gps, pick the most expensive car you can see and follow it” is specific to the problem of being lost. Second, all heuristics are not equally useful. One heuristic that many students know is “when in doubt, choose c for a question on a multiple-choice exam.” This is a dreadful strategy because many instructors intentionally randomize the order of answer choices. Another test-taking heuristic, somewhat more useful, is “look for the answer to one question somewhere else on the exam.”

You really should pay attention to the application of heuristics to test taking. Imagine that while reviewing your answers for a multiple-choice exam before turning it in, you come across a question for which you originally thought the answer was c. Upon reflection, you now think that the answer might be b. Should you change the answer to b, or should you stick with your first impression? Most people will apply the heuristic strategy to “stick with your first impression.” What they do not realize, of course, is that this is a very poor strategy (Lilienfeld et al, 2009). Most of the errors on exams come on questions that were answered wrong originally and were not changed (so they remain wrong). There are many fewer errors where we change a correct answer to an incorrect answer. And, of course, sometimes we change an incorrect answer to a correct answer. In fact, research has shown that it is more common to change a wrong answer to a right answer than vice versa (Bruno, 2001).

The belief in this poor test-taking strategy (stick with your first impression) is based on the confirmation bias (Nickerson, 1998; Wason, 1960). You first saw the confirmation bias in Module 1, but because it is so important, we will repeat the information here. People have a bias, or tendency, to notice information that confirms what they already believe. Somebody at one time told you to stick with your first impression, so when you look at the results of an exam you have taken, you will tend to notice the cases that are consistent with that belief. That is, you will notice the cases in which you originally had an answer correct and changed it to the wrong answer. You tend not to notice the other two important (and more common) cases, changing an answer from wrong to right, and leaving a wrong answer unchanged.

Because heuristics by definition do not guarantee a correct solution to a problem, mistakes are bound to occur when we employ them. A poor choice of a specific heuristic will lead to an even higher likelihood of making an error.

algorithm : a step-by-step procedure that guarantees a correct solution to a problem

problem solving heuristic : a shortcut strategy that we use to solve problems. Although they are easy to use, they do not guarantee correct judgments and solutions

confirmation bias : people’s tendency to notice information that confirms what they already believe

An Effective Problem-Solving Sequence

You may be left with a big question: If algorithms are hard to use and heuristics often don’t work, how am I supposed to solve problems? Robert Sternberg (1996), as part of his theory of what makes people successfully intelligent (Module 8) described a problem-solving sequence that has been shown to work rather well:

Identify the existence of a problem. In school, problem identification is often easy; problems that you encounter in math classes, for example, are conveniently labeled as problems for you. Outside of school, however, realizing that you have a problem is a key difficulty that you must get past in order to begin solving it. You must be very sensitive to the symptoms that indicate a problem.
Define the problem. Suppose you realize that you have been having many headaches recently. Very likely, you would identify this as a problem. If you define the problem as “headaches,” the solution would probably be to take aspirin or ibuprofen or some other anti-inflammatory medication. If the headaches keep returning, however, you have not really solved the problem—likely because you have mistaken a symptom for the problem itself. Instead, you must find the root cause of the headaches. Stress might be the real problem. For you to successfully solve many problems it may be necessary for you to overcome your fixations and represent the problems differently. One specific strategy that you might find useful is to try to define the problem from someone else’s perspective. How would your parents, spouse, significant other, doctor, etc. define the problem? Somewhere in these different perspectives may lurk the key definition that will allow you to find an easier and permanent solution.
Formulate strategy. Now it is time to begin planning exactly how the problem will be solved. Is there an algorithm or heuristic available for you to use? Remember, heuristics by their very nature guarantee that occasionally you will not be able to solve the problem. One point to keep in mind is that you should look for long-range solutions, which are more likely to address the root cause of a problem than short-range solutions.
Represent and organize information. Similar to the way that the problem itself can be defined, or represented in multiple ways, information within the problem is open to different interpretations. Suppose you are studying for a big exam. You have chapters from a textbook and from a supplemental reader, along with lecture notes that all need to be studied. How should you (represent and) organize these materials? Should you separate them by type of material (text versus reader versus lecture notes), or should you separate them by topic? To solve problems effectively, you must learn to find the most useful representation and organization of information.
Allocate resources. This is perhaps the simplest principle of the problem solving sequence, but it is extremely difficult for many people. First, you must decide whether time, money, skills, effort, goodwill, or some other resource would help to solve the problem Then, you must make the hard choice of deciding which resources to use, realizing that you cannot devote maximum resources to every problem. Very often, the solution to problem is simply to change how resources are allocated (for example, spending more time studying in order to improve grades).
Monitor and evaluate solutions. Pay attention to the solution strategy while you are applying it. If it is not working, you may be able to select another strategy. Another fact you should realize about problem solving is that it never does end. Solving one problem frequently brings up new ones. Good monitoring and evaluation of your problem solutions can help you to anticipate and get a jump on solving the inevitable new problems that will arise.

Please note that this as an effective problem-solving sequence, not the effective problem solving sequence. Just as you can become fixated and end up representing the problem incorrectly or trying an inefficient solution, you can become stuck applying the problem-solving sequence in an inflexible way. Clearly there are problem situations that can be solved without using these skills in this order.

Additionally, many real-world problems may require that you go back and redefine a problem several times as the situation changes (Sternberg et al. 2000). For example, consider the problem with Mary’s daughter one last time. At first, Mary did represent the problem as one of defiance. When her early strategy of pleading and threatening punishment was unsuccessful, Mary began to observe her daughter more carefully. She noticed that, indeed, her daughter’s attention would be drawn by an irresistible distraction or book. Fresh with a re-representation of the problem, she began a new solution strategy. She began to remind her daughter every few minutes to stay on task and remind her that if she is ready before it is time to leave, she may return to the book or other distracting object at that time. Fortunately, this strategy was successful, so Mary did not have to go back and redefine the problem again.

Pick one or two of the problems that you listed when you first started studying this section and try to work out the steps of Sternberg’s problem solving sequence for each one.

a mental representation of a category of things in the world

an assumption about the truth of something that is not stated. Inferences come from our prior knowledge and experience, and from logical reasoning

knowledge about one’s own cognitive processes; thinking about your thinking

individuals who are less competent tend to overestimate their abilities more than individuals who are more competent do

Thinking like a scientist in your everyday life for the purpose of drawing correct conclusions. It entails skepticism; an ability to identify biases, distortions, omissions, and assumptions; and excellent deductive and inductive reasoning, and problem solving skills.

a way of thinking in which you refrain from drawing a conclusion or changing your mind until good evidence has been provided

an inclination, tendency, leaning, or prejudice

a type of reasoning in which the conclusion is guaranteed to be true any time the statements leading up to it are true

a set of statements in which the beginning statements lead to a conclusion

an argument for which true beginning statements guarantee that the conclusion is true

a type of reasoning in which we make judgments about likelihood from sets of evidence

an inductive argument in which the beginning statements lead to a conclusion that is probably true

fast, automatic, and emotional thinking

slow, effortful, and logical thinking

a shortcut strategy that we use to make judgments and solve problems. Although they are easy to use, they do not guarantee correct judgments and solutions

udging the frequency or likelihood of some event type according to how easily examples of the event can be called to mind (i.e., how available they are to memory)

judging the likelihood that something is a member of a category on the basis of how much it resembles a typical category member (i.e., how representative it is of the category)

a situation in which we are in an initial state, have a desired goal state, and there is a number of possible intermediate states (i.e., there is no obvious way to get from the initial to the goal state)

noticing, comprehending and forming a mental conception of a problem

when a problem solver gets stuck looking at a problem a particular way and cannot change his or her representation of it (or his or her intended solution strategy)

a specific type of fixation in which a problem solver cannot think of a new use for an object that already has a function

a specific type of fixation in which a problem solver gets stuck using the same solution strategy that has been successful in the past

a sudden realization of a solution to a problem

a step-by-step procedure that guarantees a correct solution to a problem

The tendency to notice and pay attention to information that confirms your prior beliefs and to ignore information that disconfirms them.

a shortcut strategy that we use to solve problems. Although they are easy to use, they do not guarantee correct judgments and solutions

Introduction to Psychology Copyright © 2020 by Ken Gray; Elizabeth Arnott-Hill; and Or'Shaundra Benson is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Share This Book

6 Thinking and Intelligence

Three side by side images are shown. On the left is a person lying in the grass with a book, looking off into the distance. In the middle is a sculpture of a person sitting on rock, with chin rested on hand, and the elbow of that hand rested on knee. The third is a drawing of a person sitting cross-legged with his head resting on his hand, elbow on knee.

What is the best way to solve a problem? How does a person who has never seen or touched snow in real life develop an understanding of the concept of snow? How do young children acquire the ability to learn language with no formal instruction? Psychologists who study thinking explore questions like these and are called cognitive psychologists.

Cognitive psychologists also study intelligence. What is intelligence, and how does it vary from person to person? Are “street smarts” a kind of intelligence, and if so, how do they relate to other types of intelligence? What does an IQ test really measure? These questions and more will be explored in this chapter as you study thinking and intelligence.

In other chapters, we discussed the cognitive processes of perception, learning, and memory. In this chapter, we will focus on high-level cognitive processes. As a part of this discussion, we will consider thinking and briefly explore the development and use of language. We will also discuss problem solving and creativity before ending with a discussion of how intelligence is measured and how our biology and environments interact to affect intelligence. After finishing this chapter, you will have a greater appreciation of the higher-level cognitive processes that contribute to our distinctiveness as a species.

Learning Objectives

By the end of this section, you will be able to:

Describe cognition
Distinguish concepts and prototypes
Explain the difference between natural and artificial concepts
Describe how schemata are organized and constructed

Imagine all of your thoughts as if they were physical entities, swirling rapidly inside your mind. How is it possible that the brain is able to move from one thought to the next in an organized, orderly fashion? The brain is endlessly perceiving, processing, planning, organizing, and remembering—it is always active. Yet, you don’t notice most of your brain’s activity as you move throughout your daily routine. This is only one facet of the complex processes involved in cognition. Simply put, cognition is thinking, and it encompasses the processes associated with perception, knowledge, problem solving, judgment, language, and memory. Scientists who study cognition are searching for ways to understand how we integrate, organize, and utilize our conscious cognitive experiences without being aware of all of the unconscious work that our brains are doing (for example, Kahneman, 2011).

Upon waking each morning, you begin thinking—contemplating the tasks that you must complete that day. In what order should you run your errands? Should you go to the bank, the cleaners, or the grocery store first? Can you get these things done before you head to class or will they need to wait until school is done? These thoughts are one example of cognition at work. Exceptionally complex, cognition is an essential feature of human consciousness, yet not all aspects of cognition are consciously experienced.

Cognitive psychology is the field of psychology dedicated to examining how people think. It attempts to explain how and why we think the way we do by studying the interactions among human thinking, emotion, creativity, language, and problem solving, in addition to other cognitive processes. Cognitive psychologists strive to determine and measure different types of intelligence, why some people are better at problem solving than others, and how emotional intelligence affects success in the workplace, among countless other topics. They also sometimes focus on how we organize thoughts and information gathered from our environments into meaningful categories of thought, which will be discussed later.

Concepts and Prototypes

The human nervous system is capable of handling endless streams of information. The senses serve as the interface between the mind and the external environment, receiving stimuli and translating it into nerve impulses that are transmitted to the brain. The brain then processes this information and uses the relevant pieces to create thoughts, which can then be expressed through language or stored in memory for future use. To make this process more complex, the brain does not gather information from external environments only. When thoughts are formed, the mind synthesizes information from emotions and memories ( Figure 7.2 ). Emotion and memory are powerful influences on both our thoughts and behaviors.

The outline of a human head is shown. There is a box containing “Information, sensations” in front of the head. An arrow from this box points to another box containing “Emotions, memories” located where the front of the person's brain would be. An arrow from this second box points to a third box containing “Thoughts” located where the back of the person's brain would be. There are two arrows coming from “Thoughts.” One arrow points back to the second box, “Emotions, memories,” and the other arrow points to a fourth box, “Behavior.”

In order to organize this staggering amount of information, the mind has developed a “file cabinet” of sorts in the mind. The different files stored in the file cabinet are called concepts. Concepts are categories or groupings of linguistic information, images, ideas, or memories, such as life experiences. Concepts are, in many ways, big ideas that are generated by observing details, and categorizing and combining these details into cognitive structures. You use concepts to see the relationships among the different elements of your experiences and to keep the information in your mind organized and accessible.

Concepts are informed by our semantic memory (you will learn more about semantic memory in a later chapter) and are present in every aspect of our lives; however, one of the easiest places to notice concepts is inside a classroom, where they are discussed explicitly. When you study United States history, for example, you learn about more than just individual events that have happened in America’s past. You absorb a large quantity of information by listening to and participating in discussions, examining maps, and reading first-hand accounts of people’s lives. Your brain analyzes these details and develops an overall understanding of American history. In the process, your brain gathers details that inform and refine your understanding of related concepts like democracy, power, and freedom.

Concepts can be complex and abstract, like justice, or more concrete, like types of birds. In psychology, for example, Piaget’s stages of development are abstract concepts. Some concepts, like tolerance, are agreed upon by many people because they have been used in various ways over many years. Other concepts, like the characteristics of your ideal friend or your family’s birthday traditions, are personal and individualized. In this way, concepts touch every aspect of our lives, from our many daily routines to the guiding principles behind the way governments function.

Another technique used by your brain to organize information is the identification of prototypes for the concepts you have developed. A prototype is the best example or representation of a concept. For example, what comes to your mind when you think of a dog? Most likely your early experiences with dogs will shape what you imagine. If your first pet was a Golden Retriever, there is a good chance that this would be your prototype for the category of dogs.

Natural and Artificial Concepts

In psychology, concepts can be divided into two categories, natural and artificial. Natural concepts are created “naturally” through your experiences and can be developed from either direct or indirect experiences. For example, if you live in Essex Junction, Vermont, you have probably had a lot of direct experience with snow. You’ve watched it fall from the sky, you’ve seen lightly falling snow that barely covers the windshield of your car, and you’ve shoveled out 18 inches of fluffy white snow as you’ve thought, “This is perfect for skiing.” You’ve thrown snowballs at your best friend and gone sledding down the steepest hill in town. In short, you know snow. You know what it looks like, smells like, tastes like, and feels like. If, however, you’ve lived your whole life on the island of Saint Vincent in the Caribbean, you may never have actually seen snow, much less tasted, smelled, or touched it. You know snow from the indirect experience of seeing pictures of falling snow—or from watching films that feature snow as part of the setting. Either way, snow is a natural concept because you can construct an understanding of it through direct observations, experiences with snow, or indirect knowledge (such as from films or books) ( Figure 7.3 ).

Photograph A shows a snow covered landscape with the sun shining over it. Photograph B shows a sphere shaped object perched atop the corner of a cube shaped object. There is also a triangular object shown.

An artificial concept , on the other hand, is a concept that is defined by a specific set of characteristics. Various properties of geometric shapes, like squares and triangles, serve as useful examples of artificial concepts. A triangle always has three angles and three sides. A square always has four equal sides and four right angles. Mathematical formulas, like the equation for area (length × width), are artificial concepts defined by specific sets of characteristics that are always the same. Artificial concepts can enhance the understanding of a topic by building on one another. For example, before learning the concept of “area of a square” (and the formula to find it), you must understand what a square is. Once the concept of “area of a square” is understood, an understanding of area for other geometric shapes can be built upon the original understanding of area. The use of artificial concepts to define an idea is crucial to communicating with others and engaging in complex thought. According to Goldstone and Kersten (2003), concepts act as building blocks and can be connected in countless combinations to create complex thoughts.

A schema is a mental construct consisting of a cluster or collection of related concepts (Bartlett, 1932). There are many different types of schemata, and they all have one thing in common: schemata are a method of organizing information that allows the brain to work more efficiently. When a schema is activated, the brain makes immediate assumptions about the person or object being observed.

There are several types of schemata. A role schema makes assumptions about how individuals in certain roles will behave (Callero, 1994). For example, imagine you meet someone who introduces himself as a firefighter. When this happens, your brain automatically activates the “firefighter schema” and begins making assumptions that this person is brave, selfless, and community-oriented. Despite not knowing this person, already you have unknowingly made judgments about him. Schemata also help you fill in gaps in the information you receive from the world around you. While schemata allow for more efficient information processing, there can be problems with schemata, regardless of whether they are accurate: Perhaps this particular firefighter is not brave, he just works as a firefighter to pay the bills while studying to become a children’s librarian.

An event schema , also known as a cognitive script , is a set of behaviors that can feel like a routine. Think about what you do when you walk into an elevator ( Figure 7.4 ). First, the doors open and you wait to let exiting passengers leave the elevator car. Then, you step into the elevator and turn around to face the doors, looking for the correct button to push. You never face the back of the elevator, do you? And when you’re riding in a crowded elevator and you can’t face the front, it feels uncomfortable, doesn’t it? Interestingly, event schemata can vary widely among different cultures and countries. For example, while it is quite common for people to greet one another with a handshake in the United States, in Tibet, you greet someone by sticking your tongue out at them, and in Belize, you bump fists (Cairns Regional Council, n.d.)

A crowded elevator is shown. There are many people standing close to one another.

Because event schemata are automatic, they can be difficult to change. Imagine that you are driving home from work or school. This event schema involves getting in the car, shutting the door, and buckling your seatbelt before putting the key in the ignition. You might perform this script two or three times each day. As you drive home, you hear your phone’s ring tone. Typically, the event schema that occurs when you hear your phone ringing involves locating the phone and answering it or responding to your latest text message. So without thinking, you reach for your phone, which could be in your pocket, in your bag, or on the passenger seat of the car. This powerful event schema is informed by your pattern of behavior and the pleasurable stimulation that a phone call or text message gives your brain. Because it is a schema, it is extremely challenging for us to stop reaching for the phone, even though we know that we endanger our own lives and the lives of others while we do it (Neyfakh, 2013) ( Figure 7.5 ).

A person’s right hand is holding a cellular phone. The person is in the driver’s seat of an automobile while on the road.

Remember the elevator? It feels almost impossible to walk in and not face the door. Our powerful event schema dictates our behavior in the elevator, and it is no different with our phones. Current research suggests that it is the habit, or event schema, of checking our phones in many different situations that make refraining from checking them while driving especially difficult (Bayer & Campbell, 2012). Because texting and driving has become a dangerous epidemic in recent years, psychologists are looking at ways to help people interrupt the “phone schema” while driving. Event schemata like these are the reason why many habits are difficult to break once they have been acquired. As we continue to examine thinking, keep in mind how powerful the forces of concepts and schemata are to our understanding of the world.

Define language and demonstrate familiarity with the components of language
Understand the development of language
Explain the relationship between language and thinking

Language is a communication system that involves using words and systematic rules to organize those words to transmit information from one individual to another. While language is a form of communication, not all communication is language. Many species communicate with one another through their postures, movements, odors, or vocalizations. This communication is crucial for species that need to interact and develop social relationships with their conspecifics. However, many people have asserted that it is language that makes humans unique among all of the animal species (Corballis & Suddendorf, 2007; Tomasello & Rakoczy, 2003). This section will focus on what distinguishes language as a special form of communication, how the use of language develops, and how language affects the way we think.

Components of Language

Language, be it spoken, signed, or written, has specific components: a lexicon and grammar. Lexicon refers to the words of a given language. Thus, lexicon is a language’s vocabulary. Grammar refers to the set of rules that are used to convey meaning through the use of the lexicon (Fernández & Cairns, 2011). For instance, English grammar dictates that most verbs receive an “-ed” at the end to indicate past tense.

Words are formed by combining the various phonemes that make up the language. A phoneme (e.g., the sounds “ah” vs. “eh”) is a basic sound unit of a given language, and different languages have different sets of phonemes. Phonemes are combined to form morphemes , which are the smallest units of language that convey some type of meaning (e.g., “I” is both a phoneme and a morpheme). We use semantics and syntax to construct language. Semantics and syntax are part of a language’s grammar. Semantics refers to the process by which we derive meaning from morphemes and words. Syntax refers to the way words are organized into sentences (Chomsky, 1965; Fernández & Cairns, 2011).

We apply the rules of grammar to organize the lexicon in novel and creative ways, which allow us to communicate information about both concrete and abstract concepts. We can talk about our immediate and observable surroundings as well as the surface of unseen planets. We can share our innermost thoughts, our plans for the future, and debate the value of a college education. We can provide detailed instructions for cooking a meal, fixing a car, or building a fire. Through our use of words and language, we are able to form, organize, and express ideas, schema, and artificial concepts.

Language Development

Given the remarkable complexity of a language, one might expect that mastering a language would be an especially arduous task; indeed, for those of us trying to learn a second language as adults, this might seem to be true. However, young children master language very quickly with relative ease. B. F. Skinner (1957) proposed that language is learned through reinforcement. Noam Chomsky (1965) criticized this behaviorist approach, asserting instead that the mechanisms underlying language acquisition are biologically determined. The use of language develops in the absence of formal instruction and appears to follow a very similar pattern in children from vastly different cultures and backgrounds. It would seem, therefore, that we are born with a biological predisposition to acquire a language (Chomsky, 1965; Fernández & Cairns, 2011). Moreover, it appears that there is a critical period for language acquisition, such that this proficiency at acquiring language is maximal early in life; generally, as people age, the ease with which they acquire and master new languages diminishes (Johnson & Newport, 1989; Lenneberg, 1967; Singleton, 1995).

Children begin to learn about language from a very early age ( Table 7.1 ). In fact, it appears that this is occurring even before we are born. Newborns show a preference for their mother’s voice and appear to be able to discriminate between the language spoken by their mother and other languages. Babies are also attuned to the languages being used around them and show preferences for videos of faces that are moving in synchrony with the audio of spoken language versus videos that do not synchronize with the audio (Blossom & Morgan, 2006; Pickens, 1994; Spelke & Cortelyou, 1981).

DIG DEEPER: The Case of Genie

In the fall of 1970, a social worker in the Los Angeles area found a 13-year-old girl who was being raised in extremely neglectful and abusive conditions. The girl, who came to be known as Genie, had lived most of her life tied to a potty chair or confined to a crib in a small room that was kept closed with the curtains drawn. For a little over a decade, Genie had virtually no social interaction and no access to the outside world. As a result of these conditions, Genie was unable to stand up, chew solid food, or speak (Fromkin, Krashen, Curtiss, Rigler, & Rigler, 1974; Rymer, 1993). The police took Genie into protective custody.

Genie’s abilities improved dramatically following her removal from her abusive environment, and early on, it appeared she was acquiring language—much later than would be predicted by critical period hypotheses that had been posited at the time (Fromkin et al., 1974). Genie managed to amass an impressive vocabulary in a relatively short amount of time. However, she never developed a mastery of the grammatical aspects of language (Curtiss, 1981). Perhaps being deprived of the opportunity to learn language during a critical period impeded Genie’s ability to fully acquire and use language.

You may recall that each language has its own set of phonemes that are used to generate morphemes, words, and so on. Babies can discriminate among the sounds that make up a language (for example, they can tell the difference between the “s” in vision and the “ss” in fission); early on, they can differentiate between the sounds of all human languages, even those that do not occur in the languages that are used in their environments. However, by the time that they are about 1 year old, they can only discriminate among those phonemes that are used in the language or languages in their environments (Jensen, 2011; Werker & Lalonde, 1988; Werker & Tees, 1984).

After the first few months of life, babies enter what is known as the babbling stage, during which time they tend to produce single syllables that are repeated over and over. As time passes, more variations appear in the syllables that they produce. During this time, it is unlikely that the babies are trying to communicate; they are just as likely to babble when they are alone as when they are with their caregivers (Fernández & Cairns, 2011). Interestingly, babies who are raised in environments in which sign language is used will also begin to show babbling in the gestures of their hands during this stage (Petitto, Holowka, Sergio, Levy, & Ostry, 2004).

Generally, a child’s first word is uttered sometime between the ages of 1 year to 18 months, and for the next few months, the child will remain in the “one word” stage of language development. During this time, children know a number of words, but they only produce one-word utterances. The child’s early vocabulary is limited to familiar objects or events, often nouns. Although children in this stage only make one-word utterances, these words often carry larger meaning (Fernández & Cairns, 2011). So, for example, a child saying “cookie” could be identifying a cookie or asking for a cookie.

As a child’s lexicon grows, she begins to utter simple sentences and to acquire new vocabulary at a very rapid pace. In addition, children begin to demonstrate a clear understanding of the specific rules that apply to their language(s). Even the mistakes that children sometimes make provide evidence of just how much they understand about those rules. This is sometimes seen in the form of overgeneralization . In this context, overgeneralization refers to an extension of a language rule to an exception to the rule. For example, in English, it is usually the case that an “s” is added to the end of a word to indicate plurality. For example, we speak of one dog versus two dogs. Young children will overgeneralize this rule to cases that are exceptions to the “add an s to the end of the word” rule and say things like “those two gooses” or “three mouses.” Clearly, the rules of the language are understood, even if the exceptions to the rules are still being learned (Moskowitz, 1978).

Language and Thought

When we speak one language, we agree that words are representations of ideas, people, places, and events. The given language that children learn is connected to their culture and surroundings. But can words themselves shape the way we think about things? Psychologists have long investigated the question of whether language shapes thoughts and actions, or whether our thoughts and beliefs shape our language. Two researchers, Edward Sapir and Benjamin Lee Whorf began this investigation in the 1940s. They wanted to understand how the language habits of a community encourage members of that community to interpret language in a particular manner (Sapir, 1941/1964). Sapir and Whorf proposed that language determines thought. For example, in some languages, there are many different words for love. However, in English, we use the word love for all types of love. Does this affect how we think about love depending on the language that we speak (Whorf, 1956)? Researchers have since identified this view as too absolute, pointing out a lack of empiricism behind what Sapir and Whorf proposed (Abler, 2013; Boroditsky, 2011; van Troyer, 1994). Today, psychologists continue to study and debate the relationship between language and thought.

Describe problem solving strategies
Define algorithm and heuristic
Explain some common roadblocks to effective problem solving and decision making

People face problems every day—usually, multiple problems throughout the day. Sometimes these problems are straightforward: To double a recipe for pizza dough, for example, all that is required is that each ingredient in the recipe is doubled. Sometimes, however, the problems we encounter are more complex. For example, say you have a work deadline, and you must mail a printed copy of a report to your supervisor by the end of the business day. The report is time-sensitive and must be sent overnight. You finished the report last night, but your printer will not work today. What should you do? First, you need to identify the problem and then apply a strategy for solving the problem.

Problem-Solving Strategies

A problem-solving strategy is a plan of action used to find a solution. Different strategies have different action plans associated with them ( Table 7.2 ). For example, a well-known strategy is trial and error . The old adage, “If at first, you don’t succeed, try, try again” describes trial and error. In terms of your broken printer, you could try checking the ink levels, and if that doesn’t work, you could check to make sure the paper tray isn’t jammed. Or maybe the printer isn’t actually connected to your laptop. When using trial and error, you would continue to try different solutions until you solved your problem. Although trial and error is not typically one of the most time-efficient strategies, it is a commonly used one.

When one is faced with too much information
When the time to make a decision is limited
When the decision to be made is unimportant
When there is access to very little information to use in making the decision
When an appropriate heuristic happens to come to mind in the same moment

Working backward is a useful heuristic in which you begin solving the problem by focusing on the end result. Consider this example: You live in Washington, D.C., and have been invited to a wedding at 4 PM on Saturday in Philadelphia. Knowing that Interstate 95 tends to back up any day of the week, you need to plan your route and time your departure accordingly. If you want to be at the wedding service by 3:30 PM, and it takes 2.5 hours to get to Philadelphia without traffic, what time should you leave your house? You use the working backward heuristic to plan the events of your day on a regular basis, probably without even thinking about it.

Another useful heuristic is the practice of accomplishing a large goal or task by breaking it into a series of smaller steps. Students often use this common method to complete a large research project or a long essay for school. For example, students typically brainstorm, develop a thesis or main topic, research the chosen topic, organize their information into an outline, write a rough draft, revise and edit the rough draft, develop a final draft, organize the references list, and proofread their work before turning in the project. The large task becomes less overwhelming when it is broken down into a series of small steps.

EVERYDAY CONNECTION: Solving Puzzles

Problem-solving abilities can improve with practice. Many people challenge themselves every day with puzzles and other mental exercises to sharpen their problem-solving skills. Sudoku puzzles appear daily in most newspapers. Typically, a sudoku puzzle is a 9×9 grid. The simple sudoku below ( Figure 7.7 ) is a 4×4 grid. To solve the puzzle, fill in the empty boxes with a single digit: 1, 2, 3, or 4. Here are the rules: The numbers must total 10 in each bolded box, each row, and each column; however, each digit can only appear once in a bolded box, row, and column. Time yourself as you solve this puzzle and compare your time with a classmate.

A four column by four row Sudoku puzzle is shown. The top left cell contains the number 3. The top right cell contains the number 2. The bottom right cell contains the number 1. The bottom left cell contains the number 4. The cell at the intersection of the second row and the second column contains the number 4. The cell to the right of that contains the number 1. The cell below the cell containing the number 1 contains the number 2. The cell to the left of the cell containing the number 2 contains the number 3.

Here is another popular type of puzzle ( Figure 7.8 ) that challenges your spatial reasoning skills. Connect all nine dots with four connecting straight lines without lifting your pencil from the paper:

A square shaped outline contains three rows and three columns of dots with equal space between them.

Take a look at the “Puzzling Scales” logic puzzle below ( Figure 7.9 ). Sam Loyd, a well-known puzzle master, created and refined countless puzzles throughout his lifetime (Cyclopedia of Puzzles, n.d.).

A puzzle involving a scale is shown. At the top of the figure it reads: “Sam Loyds Puzzling Scales.” The first row of the puzzle shows a balanced scale with 3 blocks and a top on the left and 12 marbles on the right. Below this row it reads: “Since the scales now balance.” The next row of the puzzle shows a balanced scale with just the top on the left, and 1 block and 8 marbles on the right. Below this row it reads: “And balance when arranged this way.” The third row shows an unbalanced scale with the top on the left side, which is much lower than the right side. The right side is empty. Below this row it reads: “Then how many marbles will it require to balance with that top?”

Functional fixedness is a type of mental set where you cannot perceive an object being used for something other than what it was designed for. Duncker (1945) conducted foundational research on functional fixedness. He created an experiment in which participants were given a candle, a book of matches, and a box of thumbtacks. They were instructed to use those items to attach the candle to the wall so that it did not drip wax onto the table below. Participants had to use functional fixedness to solve the problem ( Figure 7.10 ). During the Apollo 13 mission to the moon, NASA engineers at Mission Control had to overcome functional fixedness to save the lives of the astronauts aboard the spacecraft. An explosion in a module of the spacecraft damaged multiple systems. The astronauts were in danger of being poisoned by rising levels of carbon dioxide because of problems with the carbon dioxide filters. The engineers found a way for the astronauts to use spare plastic bags, tape, and air hoses to create a makeshift air filter, which saved the lives of the astronauts.

Figure a shows a book of matches, a box of thumbtacks, and a candle. Figure b shows the candle standing in the box that held the thumbtacks. A thumbtack attaches the box holding the candle to the wall.

The confirmation bias is the tendency to focus on information that confirms your existing beliefs. For example, if you think that your professor is not very nice, you notice all of the instances of rude behavior exhibited by the professor while ignoring the countless pleasant interactions he is involved in on a daily basis. Hindsight bias leads you to believe that the event you just experienced was predictable, even though it really wasn’t. In other words, you knew all along that things would turn out the way they did. Representative bias describes a faulty way of thinking, in which you unintentionally stereotype someone or something; for example, you may assume that your professors spend their free time reading books and engaging in intellectual conversation because the idea of them spending their time playing volleyball or visiting an amusement park does not fit in with your stereotypes of professors.

Were you able to determine how many marbles are needed to balance the scales in Figure 7.9 ? You need nine. Were you able to solve the problems in Figure 7.7 and Figure 7.8 ? Here are the answers ( Figure 7.11 ).

The first puzzle is a Sudoku grid of 16 squares (4 rows of 4 squares) is shown. Half of the numbers were supplied to start the puzzle and are colored blue, and half have been filled in as the puzzle’s solution and are colored red. The numbers in each row of the grid, left to right, are as follows. Row 1: blue 3, red 1, red 4, blue 2. Row 2: red 2, blue 4, blue 1, red 3. Row 3: red 1, blue 3, blue 2, red 4. Row 4: blue 4, red 2, red 3, blue 1.The second puzzle consists of 9 dots arranged in 3 rows of 3 inside of a square. The solution, four straight lines made without lifting the pencil, is shown in a red line with arrows indicating the direction of movement. In order to solve the puzzle, the lines must extend beyond the borders of the box. The four connecting lines are drawn as follows. Line 1 begins at the top left dot, proceeds through the middle and right dots of the top row, and extends to the right beyond the border of the square. Line 2 extends from the end of line 1, through the right dot of the horizontally centered row, through the middle dot of the bottom row, and beyond the square’s border ending in the space beneath the left dot of the bottom row. Line 3 extends from the end of line 2 upwards through the left dots of the bottom, middle, and top rows. Line 4 extends from the end of line 3 through the middle dot in the middle row and ends at the right dot of the bottom row.

Define intelligence
Explain the triarchic theory of intelligence
Identify the difference between intelligence theories
Explain emotional intelligence
Define creativity

Classifying Intelligence

What exactly is intelligence? The way that researchers have defined the concept of intelligence has been modified many times since the birth of psychology. British psychologist Charles Spearman believed intelligence consisted of one general factor, called g , which could be measured and compared among individuals. Spearman focused on the commonalities among various intellectual abilities and de-emphasized what made each unique. Long before modern psychology developed, however, ancient philosophers, such as Aristotle, held a similar view (Cianciolo & Sternberg, 2004).

Other psychologists believe that instead of a single factor, intelligence is a collection of distinct abilities. In the 1940s, Raymond Cattell proposed a theory of intelligence that divided general intelligence into two components: crystallized intelligence and fluid intelligence (Cattell, 1963). Crystallized intelligence is characterized as acquired knowledge and the ability to retrieve it. When you learn, remember, and recall information, you are using crystallized intelligence. You use crystallized intelligence all the time in your coursework by demonstrating that you have mastered the information covered in the course. Fluid intelligence encompasses the ability to see complex relationships and solve problems. Navigating your way home after being detoured onto an unfamiliar route because of road construction would draw upon your fluid intelligence. Fluid intelligence helps you tackle complex, abstract challenges in your daily life, whereas crystallized intelligence helps you overcome concrete, straightforward problems (Cattell, 1963).

Other theorists and psychologists believe that intelligence should be defined in more practical terms. For example, what types of behaviors help you get ahead in life? Which skills promote success? Think about this for a moment. Being able to recite all 45 presidents of the United States in order is an excellent party trick, but will knowing this make you a better person?

Robert Sternberg developed another theory of intelligence, which he titled the triarchic theory of intelligence because it sees intelligence as comprised of three parts (Sternberg, 1988): practical, creative, and analytical intelligence ( Figure 7.12 ).

Three boxes are arranged in a triangle. The top box contains “Analytical intelligence; academic problem solving and computation.” There is a line with arrows on both ends connecting this box to another box containing “Practical intelligence; street smarts and common sense.” Another line with arrows on both ends connects this box to another box containing “Creative intelligence; imaginative and innovative problem solving.” Another line with arrows on both ends connects this box to the first box described, completing the triangle.

Practical intelligence , as proposed by Sternberg, is sometimes compared to “street smarts.” Being practical means you find solutions that work in your everyday life by applying knowledge based on your experiences. This type of intelligence appears to be separate from the traditional understanding of IQ; individuals who score high in practical intelligence may or may not have comparable scores in creative and analytical intelligence (Sternberg, 1988).

Analytical intelligence is closely aligned with academic problem solving and computations. Sternberg says that analytical intelligence is demonstrated by an ability to analyze, evaluate, judge, compare, and contrast. When reading a classic novel for a literature class, for example, it is usually necessary to compare the motives of the main characters of the book or analyze the historical context of the story. In a science course such as anatomy, you must study the processes by which the body uses various minerals in different human systems. In developing an understanding of this topic, you are using analytical intelligence. When solving a challenging math problem, you would apply analytical intelligence to analyze different aspects of the problem and then solve it section by section.

Creative intelligence is marked by inventing or imagining a solution to a problem or situation. Creativity in this realm can include finding a novel solution to an unexpected problem or producing a beautiful work of art or a well-developed short story. Imagine for a moment that you are camping in the woods with some friends and realize that you’ve forgotten your camp coffee pot. The person in your group who figures out a way to successfully brew coffee for everyone would be credited as having higher creative intelligence.

Multiple Intelligences Theory was developed by Howard Gardner, a Harvard psychologist and former student of Erik Erikson. Gardner’s theory, which has been refined for more than 30 years, is a more recent development among theories of intelligence. In Gardner’s theory, each person possesses at least eight intelligences. Among these eight intelligences, a person typically excels in some and falters in others (Gardner, 1983). Table 7.4 describes each type of intelligence.

Gardner’s theory is relatively new and needs additional research to better establish empirical support. At the same time, his ideas challenge the traditional idea of intelligence to include a wider variety of abilities, although it has been suggested that Gardner simply relabeled what other theorists called “cognitive styles” as “intelligences” (Morgan, 1996). Furthermore, developing traditional measures of Gardner’s intelligences is extremely difficult (Furnham, 2009; Gardner & Moran, 2006; Klein, 1997).

Gardner’s inter- and intrapersonal intelligences are often combined into a single type: emotional intelligence. Emotional intelligence encompasses the ability to understand the emotions of yourself and others, show empathy, understand social relationships and cues, and regulate your own emotions and respond in culturally appropriate ways (Parker, Saklofske, & Stough, 2009). People with high emotional intelligence typically have well-developed social skills. Some researchers, including Daniel Goleman, the author of Emotional Intelligence: Why It Can Matter More than IQ , argue that emotional intelligence is a better predictor of success than traditional intelligence (Goleman, 1995). However, emotional intelligence has been widely debated, with researchers pointing out inconsistencies in how it is defined and described, as well as questioning results of studies on a subject that is difficult to measure and study empirically (Locke, 2005; Mayer, Salovey, & Caruso, 2004)

The most comprehensive theory of intelligence to date is the Cattell-Horn-Carroll (CHC) theory of cognitive abilities (Schneider & McGrew, 2018). In this theory, abilities are related and arranged in a hierarchy with general abilities at the top, broad abilities in the middle, and narrow (specific) abilities at the bottom. The narrow abilities are the only ones that can be directly measured; however, they are integrated within the other abilities. At the general level is general intelligence. Next, the broad level consists of general abilities such as fluid reasoning, short-term memory, and processing speed. Finally, as the hierarchy continues, the narrow level includes specific forms of cognitive abilities. For example, short-term memory would further break down into memory span and working memory capacity.

Intelligence can also have different meanings and values in different cultures. If you live on a small island, where most people get their food by fishing from boats, it would be important to know how to fish and how to repair a boat. If you were an exceptional angler, your peers would probably consider you intelligent. If you were also skilled at repairing boats, your intelligence might be known across the whole island. Think about your own family’s culture. What values are important for Latinx families? Italian families? In Irish families, hospitality and telling an entertaining story are marks of the culture. If you are a skilled storyteller, other members of Irish culture are likely to consider you intelligent.

Some cultures place a high value on working together as a collective. In these cultures, the importance of the group supersedes the importance of individual achievement. When you visit such a culture, how well you relate to the values of that culture exemplifies your cultural intelligence , sometimes referred to as cultural competence.

Creativity is the ability to generate, create, or discover new ideas, solutions, and possibilities. Very creative people often have intense knowledge about something, work on it for years, look at novel solutions, seek out the advice and help of other experts, and take risks. Although creativity is often associated with the arts, it is actually a vital form of intelligence that drives people in many disciplines to discover something new. Creativity can be found in every area of life, from the way you decorate your residence to a new way of understanding how a cell works.

Creativity is often assessed as a function of one’s ability to engage in divergent thinking . Divergent thinking can be described as thinking “outside the box;” it allows an individual to arrive at unique, multiple solutions to a given problem. In contrast, convergent thinking describes the ability to provide a correct or well-established answer or solution to a problem (Cropley, 2006; Gilford, 1967)

Explain how intelligence tests are developed
Describe the history of the use of IQ tests
Describe the purposes and benefits of intelligence testing

While you’re likely familiar with the term “IQ” and associate it with the idea of intelligence, what does IQ really mean? IQ stands for intelligence quotient and describes a score earned on a test designed to measure intelligence. You’ve already learned that there are many ways psychologists describe intelligence (or more aptly, intelligences). Similarly, IQ tests—the tools designed to measure intelligence—have been the subject of debate throughout their development and use.

When might an IQ test be used? What do we learn from the results, and how might people use this information? While there are certainly many benefits to intelligence testing, it is important to also note the limitations and controversies surrounding these tests. For example, IQ tests have sometimes been used as arguments in support of insidious purposes, such as the eugenics movement (Severson, 2011). The infamous Supreme Court Case, Buck v. Bell , legalized the forced sterilization of some people deemed “feeble-minded” through this type of testing, resulting in about 65,000 sterilizations ( Buck v. Bell , 274 U.S. 200; Ko, 2016). Today, only professionals trained in psychology can administer IQ tests, and the purchase of most tests requires an advanced degree in psychology. Other professionals in the field, such as social workers and psychiatrists, cannot administer IQ tests. In this section, we will explore what intelligence tests measure, how they are scored, and how they were developed.

Measuring Intelligence

It seems that the human understanding of intelligence is somewhat limited when we focus on traditional or academic-type intelligence. How then, can intelligence be measured? And when we measure intelligence, how do we ensure that we capture what we’re really trying to measure (in other words, that IQ tests function as valid measures of intelligence)? In the following paragraphs, we will explore the how intelligence tests were developed and the history of their use.

The IQ test has been synonymous with intelligence for over a century. In the late 1800s, Sir Francis Galton developed the first broad test of intelligence (Flanagan & Kaufman, 2004). Although he was not a psychologist, his contributions to the concepts of intelligence testing are still felt today (Gordon, 1995). Reliable intelligence testing (you may recall from earlier chapters that reliability refers to a test’s ability to produce consistent results) began in earnest during the early 1900s with a researcher named Alfred Binet ( Figure 7.13 ). Binet was asked by the French government to develop an intelligence test to use on children to determine which ones might have difficulty in school; it included many verbally based tasks. American researchers soon realized the value of such testing. Louis Terman, a Stanford professor, modified Binet’s work by standardizing the administration of the test and tested thousands of different-aged children to establish an average score for each age. As a result, the test was normed and standardized, which means that the test was administered consistently to a large enough representative sample of the population that the range of scores resulted in a bell curve (bell curves will be discussed later). Standardization means that the manner of administration, scoring, and interpretation of results is consistent. Norming involves giving a test to a large population so data can be collected comparing groups, such as age groups. The resulting data provide norms, or referential scores, by which to interpret future scores. Norms are not expectations of what a given group should know but a demonstration of what that group does know. Norming and standardizing the test ensures that new scores are reliable. This new version of the test was called the Stanford-Binet Intelligence Scale (Terman, 1916). Remarkably, an updated version of this test is still widely used today.

Photograph A shows a portrait of Alfred Binet. Photograph B shows six sketches of human faces. Above these faces is the label “Guide for Binet-Simon Scale. 223” The faces are arranged in three rows of two, and these rows are labeled “1, 2, and 3.” At the bottom it reads: “The psychological clinic is indebted for the loan of these cuts and those on p. 225 to the courtesy of Dr. Oliver P. Cornman, Associate Superintendent of Schools of Philadelphia, and Chairman of Committee on Backward Children Investigation. See Report of Committee, Dec. 31, 1910, appendix.”

In 1939, David Wechsler, a psychologist who spent part of his career working with World War I veterans, developed a new IQ test in the United States. Wechsler combined several subtests from other intelligence tests used between 1880 and World War I. These subtests tapped into a variety of verbal and nonverbal skills because Wechsler believed that intelligence encompassed “the global capacity of a person to act purposefully, to think rationally, and to deal effectively with his environment” (Wechsler, 1958, p. 7). He named the test the Wechsler-Bellevue Intelligence Scale (Wechsler, 1981). This combination of subtests became one of the most extensively used intelligence tests in the history of psychology. Although its name was later changed to the Wechsler Adult Intelligence Scale (WAIS) and has been revised several times, the aims of the test remain virtually unchanged since its inception (Boake, 2002). Today, there are three intelligence tests credited to Wechsler, the Wechsler Adult Intelligence Scale-fourth edition (WAIS-IV), the Wechsler Intelligence Scale for Children (WISC-V), and the Wechsler Preschool and Primary Scale of Intelligence—IV (WPPSI-IV) (Wechsler, 2012). These tests are used widely in schools and communities throughout the United States, and they are periodically normed and standardized as a means of recalibration. As a part of the recalibration process, the WISC-V was given to thousands of children across the country, and children taking the test today are compared with their same-age peers ( Figure 7.13 ).

The WISC-V is composed of 14 subtests, which comprise five indices, which then render an IQ score. The five indices are Verbal Comprehension, Visual Spatial, Fluid Reasoning, Working Memory, and Processing Speed. When the test is complete, individuals receive a score for each of the five indices and a Full Scale IQ score. The method of scoring reflects the understanding that intelligence is comprised of multiple abilities in several cognitive realms and focuses on the mental processes that the child used to arrive at his or her answers to each test item.

Interestingly, the periodic recalibrations have led to an interesting observation known as the Flynn effect. Named after James Flynn, who was among the first to describe this trend, the Flynn effect refers to the observation that each generation has a significantly higher IQ than the last. Flynn himself argues, however, that increased IQ scores do not necessarily mean that younger generations are more intelligent per se (Flynn, Shaughnessy, & Fulgham, 2012).

Ultimately, we are still left with the question of how valid intelligence tests are. Certainly, the most modern versions of these tests tap into more than verbal competencies, yet the specific skills that should be assessed in IQ testing, the degree to which any test can truly measure an individual’s intelligence, and the use of the results of IQ tests are still issues of debate (Gresham & Witt, 1997; Flynn, Shaughnessy, & Fulgham, 2012; Richardson, 2002; Schlinger, 2003).

The Bell Curve

The results of intelligence tests follow the bell curve, a graph in the general shape of a bell. When the bell curve is used in psychological testing, the graph demonstrates a normal distribution of a trait, in this case, intelligence, in the human population. Many human traits naturally follow the bell curve. For example, if you lined up all your female schoolmates according to height, it is likely that a large cluster of them would be the average height for an American woman: 5’4”–5’6”. This cluster would fall in the center of the bell curve, representing the average height for American women ( Figure 7.14 ). There would be fewer women who stand closer to 4’11”. The same would be true for women of above-average height: those who stand closer to 5’11”. The trick to finding a bell curve in nature is to use a large sample size. Without a large sample size, it is less likely that the bell curve will represent the wider population. A representative sample is a subset of the population that accurately represents the general population. If, for example, you measured the height of the women in your classroom only, you might not actually have a representative sample. Perhaps the women’s basketball team wanted to take this course together, and they are all in your class. Because basketball players tend to be taller than average, the women in your class may not be a good representative sample of the population of American women. But if your sample included all the women at your school, it is likely that their heights would form a natural bell curve.

A graph of a bell curve is labeled “Height of U.S. Women.” The x axis is labeled “Height” and the y axis is labeled “Frequency.” Between the heights of five feet tall and five feet and five inches tall, the frequency rises to a curved peak, then begins dropping off at the same rate until it hits five feet ten inches tall.

The same principles apply to intelligence test scores. Individuals earn a score called an intelligence quotient (IQ). Over the years, different types of IQ tests have evolved, but the way scores are interpreted remains the same. The average IQ score on an IQ test is 100. Standard deviations describe how data are dispersed in a population and give context to large data sets. The bell curve uses the standard deviation to show how all scores are dispersed from the average score ( Figure 7.15 ). In modern IQ testing, one standard deviation is 15 points. So a score of 85 would be described as “one standard deviation below the mean.” How would you describe a score of 115 and a score of 70? Any IQ score that falls within one standard deviation above and below the mean (between 85 and 115) is considered average, and 68% of the population has IQ scores in this range. An IQ score of 130 or above is considered a superior level.

A graph of a bell curve is labeled “Intelligence Quotient Score.” The x axis is labeled “IQ,” and the y axis is labeled “Population.” Beginning at an IQ of 60, the population rises to a curved peak at an IQ of 100 and then drops off at the same rate ending near zero at an IQ of 140.

Only 2.2% of the population has an IQ score below 70 (American Psychological Association [APA], 2013). A score of 70 or below indicates significant cognitive delays. When these are combined with major deficits in adaptive functioning, a person is diagnosed with having an intellectual disability (American Association on Intellectual and Developmental Disabilities, 2013). Formerly known as mental retardation, the accepted term now is intellectual disability, and it has four subtypes: mild, moderate, severe, and profound ( Table 7.5 ). The Diagnostic and Statistical Manual of Psychological Disorders lists criteria for each subgroup (APA, 2013).

On the other end of the intelligence spectrum are those individuals whose IQs fall into the highest ranges. Consistent with the bell curve, about 2% of the population falls into this category. People are considered gifted if they have an IQ score of 130 or higher, or superior intelligence in a particular area. Long ago, popular belief suggested that people of high intelligence were maladjusted. This idea was disproven through a groundbreaking study of gifted children. In 1921, Lewis Terman began a longitudinal study of over 1500 children with IQs over 135 (Terman, 1925). His findings showed that these children became well-educated, successful adults who were, in fact, well-adjusted (Terman & Oden, 1947). Additionally, Terman’s study showed that the subjects were above average in physical build and attractiveness, dispelling an earlier popular notion that highly intelligent people were “weaklings.” Some people with very high IQs elect to join Mensa, an organization dedicated to identifying, researching, and fostering intelligence. Members must have an IQ score in the top 2% of the population, and they may be required to pass other exams in their application to join the group.

DIG DEEPER: What’s in a Name?

In the past, individuals with IQ scores below 70 and significant adaptive and social functioning delays were diagnosed with mental retardation. When this diagnosis was first named, the title held no social stigma. In time, however, the degrading word “retard” sprang from this diagnostic term. “Retard” was frequently used as a taunt, especially among young people, until the words “mentally retarded” and “retard” became an insult. As such, the DSM-5 now labels this diagnosis as “intellectual disability.” Many states once had a Department of Mental Retardation to serve those diagnosed with such cognitive delays, but most have changed their name to the Department of Developmental Disabilities or something similar in language.

Erin Johnson’s younger brother Matthew has Down syndrome. She wrote this piece about what her brother taught her about the meaning of intelligence:

His whole life, learning has been hard. Entirely possible – just different. He has always excelled with technology – typing his thoughts was more effective than writing them or speaking them. Nothing says “leave me alone” quite like a text that reads, “Do Not Call Me Right Now.” He is fully capable of reading books up to about a third-grade level, but he didn’t love it and used to always ask others to read to him. That all changed when his nephew came along, because he willingly reads to him, and it is the most heart-swelling, smile-inducing experience I have ever had the pleasure of witnessing.

When it comes down to it, Matt can learn. He does learn. It just takes longer, and he has to work harder for it, which if we’re being honest, is not a lot of fun. He is extremely gifted in learning things he takes an interest in, and those things often seem a bit “strange” to others. But no matter. It just proves my point – he can learn. That does not mean he will learn at the same pace, or even to the same level. It also, unfortunately, does not mean he will be allotted the same opportunities to learn as many others.

Here’s the scoop. We are all wired with innate abilities to retain and apply our learning and natural curiosities and passions that fuel our desire to learn. But our abilities and curiosities may not be the same.

The world doesn’t work this way though, especially not for my brother and his counterparts. Have him read aloud a book about skunks, and you may not get a whole lot from him. But have him tell you about skunks straight out of his memory, and hold onto your hats. He can hack the school’s iPad system, but he can’t tell you how he did it. He can write out every direction for a drive to our grandparents’ home in Florida, but he can’t drive.

Society is quick to deem him disabled and use demeaning language like the r-word to describe him, but in reality, we haven’t necessarily given him opportunities to showcase the learning he can do. In my case, I can escape the need to memorize how to change the oil in my car without anyone assuming I can’t do it, or calling me names when they find out I can’t. But Matthew can’t get through a day at his job without someone assuming he needs help. He is bright. Brighter than most anyone would assume. Maybe we need to redefine what is smart.

My brother doesn’t fit in the narrow schema of intelligence that is accepted in our society. But intelligence is far more than being able to solve 525 x 62 or properly introduce yourself to another. Why can’t we assume the intelligence of someone who can recite all of a character’s lines in a movie or remember my birthday a year after I told him/her a single time? Why is it we allow a person’s diagnosis or appearance to make us not just wonder if, but entirely doubt that they are capable? Maybe we need to cut away the sides of the box we have created for people so everyone can fit.

My brother can learn. It may not be what you know. It may be knowledge you would deem unimportant. It may not follow a traditional learning trajectory. But the fact remains – he can learn. Everyone can learn. And even though it is harder for him and harder for others still, he is not a “retard.” Nobody is.

When you use the r-word, you are insinuating that an individual, whether someone with a disability or not, is unintelligent, foolish, and purposeless. This in turn tells a person with a disability that they too are unintelligent, foolish, and purposeless. Because the word was historically used to describe individuals with disabilities and twisted from its original meaning to fit a cruel new context, it is forevermore associated with people like my brother. No matter how a person looks or learns or behaves, the r-word is never a fitting term. It’s time we waved it goodbye.

Why Measure Intelligence?

The value of IQ testing is most evident in educational or clinical settings. Children who seem to be experiencing learning difficulties or severe behavioral problems can be tested to ascertain whether the child’s difficulties can be partly attributed to an IQ score that is significantly different from the mean for her age group. Without IQ testing—or another measure of intelligence—children and adults needing extra support might not be identified effectively. In addition, IQ testing is used in courts to determine whether a defendant has special or extenuating circumstances that preclude him from participating in some way in a trial. People also use IQ testing results to seek disability benefits from the Social Security Administration.

Describe how genetics and environment affect intelligence
Explain the relationship between IQ scores and socioeconomic status
Describe the difference between a learning disability and a developmental disorder

High Intelligence: Nature or Nurture?

Where does high intelligence come from? Some researchers believe that intelligence is a trait inherited from a person’s parents. Scientists who research this topic typically use twin studies to determine the heritability of intelligence. The Minnesota Study of Twins Reared Apart is one of the most well-known twin studies. In this investigation, researchers found that identical twins raised together and identical twins raised apart exhibit a higher correlation between their IQ scores than siblings or fraternal twins raised together (Bouchard, Lykken, McGue, Segal, & Tellegen, 1990). The findings from this study reveal a genetic component to intelligence ( Figure 7.15 ). At the same time, other psychologists believe that intelligence is shaped by a child’s developmental environment. If parents were to provide their children with intellectual stimuli from before they are born, it is likely that they would absorb the benefits of that stimulation, and it would be reflected in intelligence levels.

A chart shows correlations of IQs for people of varying relationships. The bottom is labeled “Percent IQ Correlation” and the left side is labeled “Relationship.” The percent IQ Correlation for relationships where no genes are shared, including adoptive parent-child pairs, similarly aged unrelated children raised together, and adoptive siblings are around 21 percent, 30 percent, and 32 percent, respectively. The percent IQ Correlation for relationships where 25 percent of genes are shared, as in half-siblings, is around 33 percent. The percent IQ Correlation for relationships where 50 percent of genes are shared, including parent-children pairs, and fraternal twins raised together, are roughly 44 percent and 62 percent, respectively. A relationship where 100 percent of genes are shared, as in identical twins raised apart, results in a nearly 80 percent IQ correlation.

The reality is that aspects of each idea are probably correct. In fact, one study suggests that although genetics seem to be in control of the level of intelligence, the environmental influences provide both stability and change to trigger manifestation of cognitive abilities (Bartels, Rietveld, Van Baal, & Boomsma, 2002). Certainly, there are behaviors that support the development of intelligence, but the genetic component of high intelligence should not be ignored. As with all heritable traits, however, it is not always possible to isolate how and when high intelligence is passed on to the next generation.

Range of Reaction is the theory that each person responds to the environment in a unique way based on his or her genetic makeup. According to this idea, your genetic potential is a fixed quantity, but whether you reach your full intellectual potential is dependent upon the environmental stimulation you experience, especially in childhood. Think about this scenario: A couple adopts a child who has average genetic intellectual potential. They raise her in an extremely stimulating environment. What will happen to the couple’s new daughter? It is likely that the stimulating environment will improve her intellectual outcomes over the course of her life. But what happens if this experiment is reversed? If a child with an extremely strong genetic background is placed in an environment that does not stimulate him: What happens? Interestingly, according to a longitudinal study of highly gifted individuals, it was found that “the two extremes of optimal and pathological experience are both represented disproportionately in the backgrounds of creative individuals”; however, those who experienced supportive family environments were more likely to report being happy (Csikszentmihalyi & Csikszentmihalyi, 1993, p. 187).

Another challenge to determining the origins of high intelligence is the confounding nature of our human social structures. It is troubling to note that some ethnic groups perform better on IQ tests than others—and it is likely that the results do not have much to do with the quality of each ethnic group’s intellect. The same is true for socioeconomic status. Children who live in poverty experience more pervasive, daily stress than children who do not worry about the basic needs of safety, shelter, and food. These worries can negatively affect how the brain functions and develops, causing a dip in IQ scores. Mark Kishiyama and his colleagues determined that children living in poverty demonstrated reduced prefrontal brain functioning comparable to children with damage to the lateral prefrontal cortex (Kishyama, Boyce, Jimenez, Perry, & Knight, 2009).

The debate around the foundations and influences on intelligence exploded in 1969 when an educational psychologist named Arthur Jensen published the article “How Much Can We Boost I.Q. and Achievement” in the Harvard Educational Review . Jensen had administered IQ tests to diverse groups of students, and his results led him to the conclusion that IQ is determined by genetics. He also posited that intelligence was made up of two types of abilities: Level I and Level II. In his theory, Level I is responsible for rote memorization, whereas Level II is responsible for conceptual and analytical abilities. According to his findings, Level I remained consistent among the human race. Level II, however, exhibited differences among ethnic groups (Modgil & Routledge, 1987). Jensen’s most controversial conclusion was that Level II intelligence is prevalent among Asians, then Caucasians, then African Americans. Robert Williams was among those who called out racial bias in Jensen’s results (Williams, 1970).

Obviously, Jensen’s interpretation of his own data caused an intense response in a nation that continued to grapple with the effects of racism (Fox, 2012). However, Jensen’s ideas were not solitary or unique; rather, they represented one of many examples of psychologists asserting racial differences in IQ and cognitive ability. In fact, Rushton and Jensen (2005) reviewed three decades worth of research on the relationship between race and cognitive ability. Jensen’s belief in the inherited nature of intelligence and the validity of the IQ test to be the truest measure of intelligence are at the core of his conclusions. If, however, you believe that intelligence is more than Levels I and II, or that IQ tests do not control for socioeconomic and cultural differences among people, then perhaps you can dismiss Jensen’s conclusions as a single window that looks out on the complicated and varied landscape of human intelligence.

In a related story, parents of African American students filed a case against the State of California in 1979, because they believed that the testing method used to identify students with learning disabilities was culturally unfair as the tests were normed and standardized using white children ( Larry P. v. Riles ). The testing method used by the state disproportionately identified African American children as mentally retarded. This resulted in many students being incorrectly classified as “mentally retarded.”

What are Learning Disabilities?

Learning disabilities are cognitive disorders that affect different areas of cognition, particularly language or reading. It should be pointed out that learning disabilities are not the same thing as intellectual disabilities. Learning disabilities are considered specific neurological impairments rather than global intellectual or developmental disabilities. A person with a language disability has difficulty understanding or using spoken language, whereas someone with a reading disability, such as dyslexia, has difficulty processing what he or she is reading.

Often, learning disabilities are not recognized until a child reaches school age. One confounding aspect of learning disabilities is that they most often affect children with average to above-average intelligence. In other words, the disability is specific to a particular area and not a measure of overall intellectual ability. At the same time, learning disabilities tend to exhibit comorbidity with other disorders, like attention-deficit hyperactivity disorder (ADHD). Anywhere between 30–70% of individuals with diagnosed cases of ADHD also have some sort of learning disability (Riccio, Gonzales, & Hynd, 1994). Let’s take a look at three examples of common learning disabilities: dysgraphia, dyslexia, and dyscalculia.

Children with dysgraphia have a learning disability that results in a struggle to write legibly. The physical task of writing with a pen and paper is extremely challenging for the person. These children often have extreme difficulty putting their thoughts down on paper (Smits-Engelsman & Van Galen, 1997). This difficulty is inconsistent with a person’s IQ. That is, based on the child’s IQ and/or abilities in other areas, a child with dysgraphia should be able to write, but can’t. Children with dysgraphia may also have problems with spatial abilities.

Students with dysgraphia need academic accommodations to help them succeed in school. These accommodations can provide students with alternative assessment opportunities to demonstrate what they know (Barton, 2003). For example, a student with dysgraphia might be permitted to take an oral exam rather than a traditional paper-and-pencil test. Treatment is usually provided by an occupational therapist, although there is some question as to how effective such treatment is (Zwicker, 2005).

Dyslexia is the most common learning disability in children. An individual with dyslexia exhibits an inability to correctly process letters. The neurological mechanism for sound processing does not work properly in someone with dyslexia. As a result, dyslexic children may not understand sound-letter correspondence. A child with dyslexia may mix up letters within words and sentences—letter reversals, such as those shown in Figure 7.17 , are a hallmark of this learning disability—or skip whole words while reading. A dyslexic child may have difficulty spelling words correctly while writing. Because of the disordered way that the brain processes letters and sounds, learning to read is a frustrating experience. Some dyslexic individuals cope by memorizing the shapes of most words, but they never actually learn to read (Berninger, 2008).

Two columns and five rows all containing the word “teapot” are shown. “Teapot” is written ten times with the letters jumbled, sometimes appearing backwards and upside down.

Dyscalculia

Dyscalculia is difficulty in learning or comprehending arithmetic. This learning disability is often first evident when children exhibit difficulty discerning how many objects are in a small group without counting them. Other symptoms may include struggling to memorize math facts, organize numbers, or fully differentiate between numerals, math symbols, and written numbers (such as “3” and “three”).

Additional Supplemental Resources

Use Google’s QuickDraw web app on your phone to quickly draw 5 things for Google’s artificially intelligent neural net. When you are done, the app will show you what it thought each of the drawings was. How does this relate to the psychological idea of concepts, prototypes, and schemas? Check out here. Works best in Chrome if used in a web browser
This article lists information about a variety of different topics relating to speech development, including how speech develops and what research is currently being done regarding speech development.
The Human intelligence site includes biographical profiles of people who have influenced the development of intelligence theory and testing, in-depth articles exploring current controversies related to human intelligence, and resources for teachers.

In 2000, psychologists Sheena Iyengar and Mark Lepper from Columbia and Stanford University published a study about the paradox of choice. This is the original journal article.
Mensa , the high IQ society, provides a forum for intellectual exchange among its members. There are members in more than 100 countries around the world. Anyone with an IQ in the top 2% of the population can join.
This test developed in the 1950s is used to refer to some kinds of behavioral tests for the presence of mind, or thought, or intelligence in putatively minded entities such as machines.
Your central “Hub” of information and products created for the network of Parent Centers serving families of children with disabilities.
How have average IQ levels changed over time? Hear James Flynn discuss the “Flynn Effect” in this Ted Talk. Closed captioning available.
We all want customized experiences and products — but when faced with 700 options, consumers freeze up. With fascinating new research, Sheena Iyengar demonstrates how businesses (and others) can improve the experience of choosing. This is the same researcher that is featured in your midterm exam.
What does an IQ Score distribution look like? Where do most people fall on an IQ Score distribution? Find out more in this video. Closed captioning available.
How do we solve problems? How can data help us to do this? Follow Amy Webb’s story of how she used algorithms to help her find her way to true love. Closed captioning available.
In this Ted-Ed video, explore some of the ways in which animals communicate, and determine whether or not this communication qualifies as language. A variety of discussion and assessment questions are included with the video (free registration is required to access the questions). Closed captioning available.
Watch this Ted-Ed video to learn more about the benefits of speaking multiple languages, including how bilingualism helps the brain to process information, strengthens the brain, and keeps the speaker more engaged in their world. A variety of discussion and assessment questions are included with the video (free registration is required to access the questions). Closed captioning available.
This video is on how your mind can amaze and betray you includes information on topics such as concepts, prototypes, problem-solving and mistakes in thinking. Closed captioning available.
This video on language includes information on topics such as the development of language, language theories, and brain areas involved in language, as well as language disorders. Closed captioning available.
This video on the controversy of intelligence includes information on topics such as theories of intelligence, emotional intelligence, and measuring intelligence. Closed captioning available.
This video on the brains vs. bias includes information on topics such as intelligence testing, testing bias, and stereotype threat. Closed captioning available.

Access for free at https://openstax.org/books/psychology-2e/pages/1-introduction

Share This Book

Learning Mathematics Problem Solving through Test Practice: a Randomized Field Experiment on a Global Scale

Intervention Study
Open access
Published: 04 February 2020
Volume 32 , pages 791–814, ( 2020 )

Cite this article

You have full access to this open access article

Francesco Avvisati 1 &
Francesca Borgonovi ORCID: orcid.org/0000-0002-6759-4515 1 , 2

8218 Accesses

7 Citations

1 Altmetric

Explore all metrics

We measure the effect of a single test practice on 15-year-old students’ ability to solve mathematics problems using large, representative samples of the schooled population in 32 countries. We exploit three unique features of the 2012 administration of the Programme for International Student Assessment (PISA), a large-scale, low-stakes international assessment. During the 2012 PISA administration, participating students were asked to sit two separate tests consisting of problem-solving tasks. Both tests included questions that covered the same internationally recognized and validated framework for mathematics assessment. Students were randomly assigned in the first, 2-h-long test to one of three test versions containing varying amounts of mathematics, reading, and science problems. We found that the amount of mathematics problems in the first test had a small positive effect on mean mathematics performance on the second test, but no effect on general reasoning and problem-solving ability. Subject-specific effects of test practice on subsequent test performance were found over both short lags (same day) and medium lags (1–7 days). The learning gains ascribed to mathematics problem-solving practice were larger for boys than for girls.

Mathematics-writing profiles for students with mathematics difficulty

Tessa L. Arsenault, Sarah R. Powell, … Danika Lang

Beyond the Standardized Assessment of Mathematical Problem Solving Competencies: From Products to Processes

Large-Scale Studies in Mathematics Education Research

Avoid common mistakes on your manuscript.

School tests have become increasingly prominent in education research and policy in many countries, sparking intense public scrutiny over their intended and unintended consequences (Coburn, Hill, & Spillane, 2016 ; Marsh, Roediger, Bjork, & Bjork, 2007 ; McNeil, 2000 ; Smith, 1991 ; Rothman, 2011 ; Tienken & Zhao, 2010 ; Vinovskis, 2008 ). High-stakes tests are used, for example, to certify students’ acquisition of skills and knowledge and to award access to selective educational and career opportunities (Madaus, 1988 ). They are also a key component of accountability systems and are used to monitor the performance of teachers and schools. Low-stakes tests, including practice tests, are used by teachers to gather information about learners’ progression and to prepare students for high-stakes test. While practice tests can promote some learning (such as learning how to allocate time efficiently on the test, or helping students memorize facts and procedures), the use of tests in today’s schools is often criticized by parents, teachers, and educators for promoting only narrow learning that is deemed to be irrelevant in the “real world,” narrowing the curriculum (Amrein & Berliner, 2002 ; Nichols & Berliner, 2007 ; Watanabe, 2007 ). Teachers who “teach to the test” and students “who learn for a test” are, in public discourse, considered to divert valuable time resources from learning (Crocco & Costigan, 2007 ; Nelson, 2013 ).

Research indicates that the impact of tests on the curriculum and classroom practice depends on the characteristics of the tests. At the level of classrooms, schools, or entire systems, it has been shown that curricular content can narrow as a result of testing, subject area knowledge can become fragmented into test-related pieces, and teachers can increase the use of teacher-centered pedagogies as a result of testing. However, tests have also led to curricular content expansion, the integration of knowledge, and more student-centered, cooperative pedagogies. The effect depends on the characteristics of the test and how tests are integrated in classroom practice (Au, 2007 ).

At the level of individual learners, findings supported by psychological research suggest that tests can be powerful ways to learn and may serve as useful aids to promote students’ ability to apply principles and procedures to new situations. In particular, there is consistent experimental evidence that administering retrieval tests after a study period enhances individuals’ ability to retain and recall information (Adesope, Trevisan, & Sundararajan, 2017 ; Carpenter & Delosh, 2006 ; Carrier & Pashler, 1992 ; Kang, McDermott, & Roediger, 2007 ; Karpicke & Blunt, 2011 ; Karpicke & Roediger, 2008 ; Roediger & Karpicke, 2006 ).

Memorization enables students to store information with the aim of subsequent reproduction (Entwistle & McCune, 2004 ). The existence of a direct effect of testing indicates that sitting a test that requires the retrieval of previously studied material can be an effective memorization strategy. However, because students are also required to master a range of learning strategies beyond memorization (OECD, 2010 ; Pritchard, 2013 ; Rubin, 1981 ; Weinstein, Ridley, Dahl, & Weber, 1989 ), the direct effect of testing, while important, is of only partial interest to education practitioners (Rohrer, Taylor & Sholar, 2010 ; Wooldridge, Bugg, McDaniel, & Liu, 2014 ).

In addition to recalling definitions and understanding key concepts, students must learn when and how to apply principles and procedures in new settings (Dirkx, Kester, & Kirschner, 2014 ). Common pedagogical styles that are considered to foster the development of students’ ability to apply principles and procedures to new situations are cognitive activation strategies, student-centered instruction and inquiry, and task-based learning (Bietenbeck, 2014 ). It is therefore important to evaluate if a testing effect can be detected when the criterial test that is used to measure outcomes requires a novel demonstration of learning (Salomon & Perkins, 1989 ).

Studies have indeed indicated that tests consisting of retrieval tasks can facilitate the encoding of new information and its successive retrieval (Pastötter & Bäuml, 2014 ; Szpunar, McDermott, & Roediger, 2008 ). A rapidly growing area of research examines the extent to which tests consisting of retrieval tasks can have transfer effects and facilitate the application of knowledge in tests consisting of new, different tasks (Barnett & Ceci, 2002 ). Recent comprehensive reviews (Carpenter, 2012 ; Pan & Rickard, 2018 ) have applied the taxonomy of transfer effects proposed by Barnett and Ceci ( 2002 ) to identify the extent to which transfer effects occur, if at all, in different contexts and settings (as defined by time, knowledge domains, and test format).

The evidence for a direct effect of testing is robust, with a large number of studies indicating that retrieval practice has a large effect on fact retention (Adesope, Trevisan, & Sundararajan, 2017 ). Proponents of transfer-appropriate processing (Morris, Bransford & Franks, 1977 ) suggest that transfer effects will be stronger if the cognitive processes invoked during the practice test are similar to those invoked during the criterial test. The evidence on transfer effects is growing but remains limited, especially when far transfer (i.e., transfer that involves extensive and/or multiple differences in context) rather than near transfer (minor differences in conditions) is considered.

The review by Pan and Rickard ( 2018 ) includes evidence from 67 published and unpublished studies on transfer effects of retrieval practice, reporting findings from 122 experiments for an overall study population of just over 10,000 individuals. The review suggests, for example, that in situations involving practice tests, transfer of learning is greatest across test formats (e.g., from a free recall test to a multiple-choice test of the same information) and from memorization to application and inference questions; it is weakest to rearranged stimulus-response items, to untested materials seen only during initial study, and to problems involving worked examples.

The limited number of studies that have been conducted on transfer effects is likely to be the result of the greater complexity of designing such studies when compared to designing studies of the direct effect of testing.

First, because in the case of far transfer, effects are expected to be smaller than for the direct effects of testing, samples have to be considerably larger and more studies have to be conducted for individual studies or for meta-analyses to have adequate power. Small effect sizes can be due to the fact that practicing the retrieval of some information may cause test takers to forget other related information leading to small estimated effects (Anderson, Bjork, & Bjork, 1994 ; Storm & Levy, 2012 ) as well as the fact that the link that exists between the material participants are exposed to in the practice test and in the criterial test is less direct in transfer-effects than that in direct-testing-effects studies (with the link being larger in near transfer than in far transfer).

Second, the identification of transfer effects would ideally require that some study participants are administered a “placebo test” in the practice session (i.e., a test that lets student practice test-taking skills, but not exercise the relevant content knowledge), and that the outcome test session include “pseudo-outcomes” (i.e., outcomes on which no effect is expected, given the hypothesized mechanism of transfer). Such a design ensures that transfer and not recall is captured by estimated effects, but also increases sample size requirements.

The present study seeks to contribute to the rich literature on testing effects using data from two field experiments embedded in the 2012 round of the Programme for International Student Assessment (PISA). PISA is an international large-scale assessment that has been administered to samples of 15-year-old students every 3 years since 2000. In each round, a minimum sample of 4500 students per participating country take part in the test and in 2012 over 60 countries participated in PISA. PISA has a global coverage, although few countries from Africa participated in the study until the most recent round in 2018. The test is administered in a wide variety of cultural, linguistic, and social contexts and stringent technical standards are implemented to ensure comparability (OECD 2014a ). Our contribution is threefold.

First, we distinguish between subject-specific learning effects and the improvement of test-taking strategies. In typical direct-testing-effects studies, the treatment consists of a retrieval session while the control is represented by a restudy session. In contrast, in our first experiment (study 1), we compared the mathematics problem-solving ability of students measured on a second test after they all sat a 2-h practice test. The difference between different groups of students was that some were asked to solve up to 1.5 h of content-relevant material (mathematics questions) while others were asked to solve as little as 0.5 h worth of content-relevant material and as much as 1.5 h of content-irrelevant material (comprising a range of science and text comprehension questions). In other words, our control group was administered a placebo test consisting mainly of content-irrelevant test questions. In a second experiment, we further compared similar experimental groups on a second test that did not contain mathematics questions, to ensure that eventual transfer effects of a greater amount of mathematics practice (or exposure) uncovered in study 1 were domain specific.

Second, by using large, representative samples from the schooled population of 15-year-olds in 32 countries worldwide, we achieved adequate power to capture small-to-medium effect sizes. Third, we exploited a field experiment based on authentic educational material to identify the effect of test practice on mathematics problem-solving performance: this has great potential to improve the ecological validity of the study and enhance the ability to draw implications for school and classroom practice.

Bjork ( 1994 , 1999 ) defined situations that promote long-term retention and transfer of knowledge at the potential expense of immediate performance as desirable difficulties. Desirable difficulties considered in the literature include distributed practice, varying conditions of practice, contextual interference, and, crucial for our study, testing (also referred to as retrieval practice) (Roediger & Karpicke, 2006 ; Roediger & Butler, 2011 ).

The studying of retrieval practice has been largely an empirical effort with the first empirical studies dating back to the early twentieth century (Abbott, 1909 ) while theoretical work aimed at understanding why a testing effect occurs and why the effect is larger in some conditions lagged somewhat behind (Roediger & Butler, 2011 ). One of the first prominent attempts to explain why testing effects occur focused on the role of exposure (Thompson et al., 1978 ). According to this theory, retrieval practice promotes learning because individuals are re-exposed to relevant material. However, subsequent empirical studies which compared test-taking to equivalent amounts of re-study, thereby ensuring similar exposure in both the test and control conditions, continued to find greater effectiveness of retrieval practice over re-study, prompting refinements in the theory (Roediger & Butler, 2011 ).

Elaborative-retrieval theory and the theory of transfer-appropriate processing define the mechanisms as to why retrieval practice can be considered to be a desirable difficulty and promote learning.

Elaborative-retrieval theory (Carpenter, 2009 ) maintains that two factors determine the testing effect: spreading activation and semantic elaboration. Spreading activation refers to the process through which retrieval practice strengthens existing retrieval routes and supports the creation of new retrieval routes. The strengthening of existing retrieval routes and the creation of new ones make it more likely that content will be successfully retrieved in the future (Roediger & Butler, 2011 ; Pyc & Rawson, 2009 ). Searching for contents in associative memory networks activates these contents as well as contents associated with it, even if the latter contents are not directly retrieved. Semantic elaboration refers to the amount of retrieval effort directed towards elaboration: greater retrieval effort corresponds to a more extensive reprocessing of the memory trace during retrieval (Roediger & Butler, 2011 ).

Elaborative retrieval theory explains the existence of a testing effect by considering that answering a series of questions demanding the recall of facts or requiring the application of a given set of principles or procedures will help students consolidate information in long-term memory (Keresztes, Kaiser, Kovács, & Racsmány, 2014 ) and promote a more integrated mental model that incorporates the target knowledge (Karpicke, 2012 ). Retrieval practice after initial encoding may also reduce the interference of competing irrelevant memories and reduce the rate of forgetting (Roediger & Karpicke, 2006 ). Elaborative retrieval theory predicts that the testing effect will be stronger the greater the amount of effort involved during retrieval.

Elaborative retrieval theory considers the effort involved during retrieval practice the factor determining why a testing effect occurs and its strength. By contrast, the theory of transfer-appropriate processing considers the match between the cognitive processes involved during the learning phase (the retrieval test) and those required during the criterial test (Morris et al., 1977 ). A greater match between the processes involved in the two phases can be expected to be associated with better performance on a final test. According to transfer-appropriate processing, the testing effect occurs because the cognitive processes involved during a practice test are more similar to those required during the final criterial test than those involved in other types of encoding activities, such as, for example, restudy (Roediger & Butler, 2011 ; Thomas & McDaniel, 2007 ). The theory of transfer-appropriate processing predicts that the testing effect will be stronger the greater the similarity between the practice and criterial tests in factors such as question format and content evoked (Roediger and Karpicke, 2006 ).

The theory of disuse (Bjork & Bjork, 1992 ) provides a comprehensive framework that can be used not only to understand why a testing effect occurs but also to make predictions about which conditions strengthen such effect. The theory distinguishes between storage strength and retrieval strength. Storage strength refers to how permanent a particular memory trace is while retrieval strength refers to how accessible such a trace is. A memory trace is high in storage strength when it is integrated with other representations and is consequently retained over the long term. A memory trace is high in retrieval strength when it is momentarily easily accessed and activated. When retrieval strength is high, short-term performance on a task is enhanced although there may be no appreciable long-term effect on performance. In fact, the theory of disuse maintains that retrieval strength is negatively associated with increments in storage strength: the easier it is to retrieve particular contents (i.e., the less semantic elaboration is involved during retrieval), the less such contents gain in storage strength (because less spreading activation occurs).

According to Bjork’s theory of disuse the strength of the testing effect may differ according to the time lag between retrieval practice and subsequent testing events, the spacing of retrieval practice sessions, the mode of retrieval delivery, i.e., whether retrieval practice consists in multiple-choice questions or constructed responses, whether corrective feedback is provided and when such feedback is provided (Adesope et al., 2017 ).

In line with theoretical predictions, empirical research identifies an advantage of retrieval practice over restudy when the retention interval (the time lag between the treatment condition and the target test event) is longer than 1 day (Karpicke & Roediger, 2008 ; Keresztes et al., 2014 ; Toppino & Cohen, 2009 ; Wheeler, Ewers, & Buonanno, 2003 ). The importance of the retention interval as a moderator of the association between practice and performance may be due to the effect of sleep on memory consolidation (Roediger et al., 2010 ) and the fact that retrieval practice may aid learning by strengthening memory traces and providing additional retrieval routes when individuals search for information in long-term memory (Keresztes et al., 2014 ).

Research also indicates that spacing retrieval practice over multiple sessions is generally more effective than the administration of a single testing session of the same duration, and that the spacing of sessions is most effective when the lag between sessions is longer and is distributed and spaced through time rather than completed in close succession (Rawson, Vaughn, & Carpenter, 2014 ; Roediger & Butler, 2011 ; Lyle et al., 2019 ).

It has been shown that tests that require participants to provide constructed responses are associated with stronger positive effects than tests that use multiple-choice response formats (McDermott, Agarwal, D’Antonio, Roediger, & McDaniel, 2014 ). Constructed responses typically require students to engage in more effortful retrieval than multiple-choice questions and effort exerted during retrieval is a factor that importantly explains the variability in the strength of the testing effect (Kang et al., 2007 ; Pyc & Rawson, 2009 ). Some even argue that, in the absence of corrective feedback, multiple-choice questions may have a negative effect on learning, particularly among individuals with low levels of baseline knowledge (Butler & Roediger, 2008 ; McDaniel & Fisher, 1991 ; Toppino & Brochin, 1989 ; Toppino & Luipersbeck, 1993 ). In multiple choice settings, individuals are exposed to a series of possible answers and if they do not know which one is correct, they may preserve, in subsequent testing events, the memory of wrong answers (Fazio, Agarwal, Marsh, & Roediger, 2010 ).

Finally, although retrieval practice has been shown to be beneficial in the absence of corrective feedback (Adesope et al., 2017 ), such feedback is associated with an increase in the benefit of testing (Agarwal, Karpicke, Kang, Roediger, & McDermott, 2008 ). The primary mechanism through which testing can promote learning is that tests promote retrieval of information. Therefore, learning effects depend on whether the correct information is retrieved. In the absence of corrective feedback, testing effects can be small or even negative (Kang et al., 2007 ).

The Present Studies

Elaborative retrieval theory and the theory of transfer-appropriate processing guided our interest in the development of the two studies that are reported in this article. Based on elaborative retrieval theory and the theory of transfer-appropriate processing, we hypothesized that what matters for performance on a criterial test is the amount of matching effort expended during practice, i.e., the amount of effort expended on tasks that relate to the same content domain. In study 1, we varied the amount of matching effort expended by varying the amount of content-relevant material in the practice test. In study 2, we varied the amount of matching effort expended by showing the effect of the same practice-test conditions on a criterial test with non-matching content.

More specifically, in study 1, we examined if performance on a criterial test consisting of mathematics tasks was enhanced when participants were involved in practice tests of equal overall length but of varying amount of mathematics content. We hypothesized that students who were exposed to a practice test containing a greater amount of content-relevant material (mathematics tasks) would perform better on a final test compared to students who were exposed to the same overall amount of testing material but who were exposed to less content-relevant material. In study 2, we examined if performance on a criterial test consisting of non-curricular problem solving tasks was associated with the amount of mathematics content in the practice test. We hypothesized that the amount of mathematics tasks contained in the practice test would not be associated with performance on the final criterial test of problem solving.

The focus on mathematics stems from the fact that mathematics is a cornerstone of curricula in secondary schools irrespective of country or educational track, and that mathematics often acts as a critical filter determining educational and career progression, course choices and occupational paths (Ma & Johnson, 2008 ).

In study 1, we set out to examine if students’ skills in solving mathematics problems could be fostered by administering tests consisting of mathematics problem-solving tasks. Similar to existing studies examining the transfer effect of testing, our target task required solving a different set of mathematics problems using principles and procedures that 15-year-olds are expected to master. However, in contrast to existing transfer-effect studies (Butler, 2010 ; Dirkx et al., 2014 ), participants in our study practiced distinct problem-solving tasks rather than recall.

We hypothesized that greater practice of mathematics problems was associated with better performance on a standardized mathematics tests. Consistent with the literature on moderators of the relationship between retrieval practice and learning, we expected that the strength of the relationship between the quantity of mathematics practice and individuals’ performance on the mathematics test would differ depending on a number of factors.

The first factor considered was the time lag between the retrieval session (practice test) and the criterial test. We hypothesized that practice would be more strongly associated with performance on the criterial test the longer the time lag between the retrieval session and the criterial test. Longer intervals have been considered to increase storage strength (Soderstrom & Bjork 2015 ) because longer intervals allow not only for the retrieval of related information but may also promote the integration of information with prior knowledge (Lyle et al. 2019 ).

The second factor considered was test-takers’ achievement in mathematics. We hypothesized that the relationship between retrieval practice and performance on the criterial test would be strongest among high-achieving students. We made this hypothesis because participants in our study did not receive corrective feedback after the retrieval practice session; therefore, in the absence of feedback, high-achieving students were more likely to retrieve correct information and to apply the correct principles and procedures during the retrieval session. Furthermore, in our study, we considered how well students performed on a mathematics test designed to assess how well they can apply principles and procedures that they learned in class over the years to solve a range of problems. High-achieving students are students for whom such principles and procedures have high storage strength and, consequently, retrieval practice could more easily stimulate accessibility.

The third factor considered was how anxious students are toward mathematics. We hypothesized that retrieval practice would be more strongly associated with performance among students with low levels of mathematics anxiety. Mathematics anxiety refers the fear of, or apprehension about, mathematics (Ashcraft & Kirk, 2001 ). The literature documents a strong negative association between mathematics anxiety and mathematics achievement (Foley et al., 2017 ). Because individuals who experience mathematics anxiety generally avoid mathematics (Ashcraft & Ridley, 2005 ), mathematics anxiety is likely to be associated with students’ level of exposure to mathematics tasks. Behavioral (Ashcraft & Kirk, 2001 ; Park, Ramirez, & Beilock, 2014 ) and fMRI (Lyons & Beilock, 2011 , 2012 ; Young, Wu, & Menon, 2012 ) studies suggest that mathematics anxiety creates worries that can deplete resources in working memory—a cognitive system responsible for short-term storage and manipulation of information (Miyake & Shah, 1999 ) that is important for learning and achieving well in math (Beilock & Carr, 2005 ; Raghubar, Barnes, & Hecht, 2010 ).

Finally, although there is no established consensus in the extent to which the effectiveness of retrieval practice varies by gender, we examined gender differences since gender gaps are a recurrent focus of the literature on mathematics performance, attitudes toward mathematics, and engagement in mathematics courses and activities (OECD, 2015 ).

In study 2, we examined whether the administration of mathematics problem-solving tasks improved students’ domain-general problem-solving skills, such as deductive reasoning, or students test-taking abilities, such as their time management, rather than specific cognitive processes involved in mathematics problem solving (such as formulating problem situations mathematically or using arithmetic procedures).

Theories of transfer-appropriate processing predict that the testing effect depends on the match between the cognitive processes involved during the final criterial test and those activated during practice tests. We therefore hypothesized that greater practice of mathematics problems (as opposed to any other kind of test practice) would be associated with better performance on a test consisting of mathematics problem solving task (study 1) but not with better performance on a test consisting of domain-general problem-solving tasks (study 2).

In both studies, we relied on comparisons between three groups created by random assignment, with each group taking a different practice test. Practice tests differed in the amount of mathematics material that they contained but all practice tests were characterized by a lack of feedback. Study 1 and study 2 were conducted at the same time, but on different participants. In this section, we describe how study participants were selected and assigned to the three groups, how materials for the tests were developed, the measures included in the data and the methods used for analyzing them and conducting the experimental comparisons.

Several characteristics contribute to the unique experimental setting employed in our study. Our sample is remarkably large, it covers a large number of countries and it is statistically representative of the wider student population in these countries. Our materials are real-world education assessment tasks developed, translated and validated by a large team of internationally recognized experts. A field trial was conducted prior to the main administration (with a different group of students) to ensure the cross-cultural validity and relevance of the test questions. Each of these features greatly enhances the external validity of our experimental method and the conclusions that can be drawn from it.

Participants: Target Population and Exclusions

Our data come from the 2012 edition of the Programme for International Student Assessment (PISA 2012), a large-scale, cross-national assessment of the mathematics, reading, and science performance of 15-year-old students. All cases used in our analyses were extracted from the public-use files for the PISA 2012 computer-based tests, which can be downloaded from http://www.oecd.org/pisa/pisaproducts/pisa2012database-downloadabledata.htm . We included all 32 national samples of countries that took part in the computer-based assessment of mathematics in our study. Footnote 1

PISA participants were selected from the population of 15-year-old students in each country according to a two-stage random sampling procedure, so that weighted samples are representative of students who are enrolled in grade 7 or above and are between 15 years and 3 months and 16 years and 2 months at the time of the assessment administration (generally referred to as 15-year-olds in this article). In the first stage, a stratified sample of schools was drawn (8321 schools in total across the 32 countries; median school sample within countries—207 schools). In the second stage, students were selected at random in each sampled school.

In study 1, we focused on students who were assigned to one of the four computer-based forms (out of 24 possible forms) containing only mathematics questions (and thus no questions from other domains). This corresponds to about three to four students per school. Since PISA assigns students to test forms at random, this subset is representative of the wider population of 15-year-old students. In total, 21,013 students included in the PISA sample Footnote 2 were assigned to take a 40-min test in mathematics on computers; they define our target population for study 1. We further excluded students from the samples used for our analysis for three reasons:

N = 89 students (in nine countries) because they used instruments that were adapted for students with special educational needs.

N = 128 students because they did not attend the first, pen-and-paper test (practice test, T1).

N = 1352 students because they did not attend the second, computer-based test session (target test, T2). Footnote 3

The final sample size for study 1 is 19,355 students.

In study 2, we used a distinct population of students ( N = 19,481), defined by those students from the same 32 countries and 8321 schools who were randomly assigned to forms containing only non-curricular “problem-solving” tasks instead of only mathematics tasks. Similar exclusion rules as for study 1 were followed.

We did not use students who were assigned to the remaining 16 forms (containing mixtures of mathematics, problem solving, and digital reading tasks), nor students who were not assigned to take a computer-based test, in any of our analyses.

Treatment Groups

In order to efficiently cover a wide range of test material in the three subjects, PISA administers, at random, different test forms to different students (OECD, 2014a ). We relied on the random allocation of test forms to different students not only to define our target population for the two studies but also to define our treatment and control groups. Table 1 outlines the PISA 2012 assessment design and its organization around 13 paper-based assessment forms.

The 13 paper-based forms differed in the amount of mathematics tasks that they contained. Three groups of forms could be defined in terms of the overall amount of their mathematics content: four forms contained only 0.5 h of testing material in mathematics, three forms contained 1 h, and the remaining six contained 1.5 h of mathematics questions. Correspondingly, in both studies, we defined three treatment arms and allocated students to these based on the test form that they were assigned. Group 1 (G1) comprised the students who had the forms with the smallest amount of mathematics content, group 2 (G2) included students who had the intermediate amount of mathematics, and group 3 (G3) included students assigned to the forms with a majority of mathematics content.

Forms were assigned to students with equal probabilities. As a result, some 30% of the students were assigned to G1 (their test contained 0.5 h of mathematics and 1.5 h of reading and/or science problems); 24% of students were assigned to G2 (their test contained 1 h of mathematics and 1 h of reading and/or science problems) and the remaining 46% of students were assigned to G3 (their test contained 1.5 h of mathematics and 0.5 h of either reading or science problems). The percentage of individuals in each group varies because there were four test forms in G1, three forms in G2, and six forms in G3.

In both studies, all participants completed two tests: a pen-and-paper test consisting of mathematics and non-mathematics tasks, T1; and a computer-based test, T2. In study 1, T2 consisted only of mathematics tasks (T2-m). In study 2, T2 consisted only of non-curricular problem-solving task (T2-ps).

All participants also completed a questionnaire in which they were asked about themselves, their family, and their school experience. Students had up to 2 h to complete the practice test T1 and up to 40 min of time to complete the criterial test T2. The questionnaire was not timed.

The sequence of operations was dictated by PISA international standards: first, students answered the pen-and-paper test (T1), then they completed the 30-min questionnaire, and finally they completed the computer-based test (T2). However, the exact timing of administration was not prescribed and countries and, to some extent, schools were free to choose the timing in order to reduce the costs of test administration (commuting costs for test administrators, etc.) and minimize the disruption to regular school activities.

The assignment of students to different forms in both tests was blind: students did not know which form they would receive before the test session. All paper-based forms had the same cover (except for the form number on the cover page) and the same front-page material, comprising a formula sheet (e.g., the Pythagorean rule) and instructions about answering formats. Until the moment students were allowed to turn pages, at the beginning of the test session and after going through the material in the front page, students were unaware of the amount of mathematics tasks included in the form. Therefore, they could not have selectively studied certain topics and domains before the practice session, based on whether these topics and domains appeared in their practice form or not. Footnote 4

The pen-and-paper and computer-based tests used test materials that were developed by an international consortium in collaboration with multi-disciplinary expert groups and under the supervision of the PISA governing board, a body of the Organisation for Economic Co-Operation and Development (OECD) composed of representatives of participating countries appointed by their respective governments. Examples of tasks included in the paper- and computer-based tests are available in the Supplementary Material .

In both studies, the practice test consisted of a mathematics section of varying length (see above, “Treatment groups”) and of a reading (text comprehension) and/or a science section; the total length of the practice test was, in all cases, of 2 h. The number of mathematics questions included in T1 varied between 12 questions for test forms in group 1 (see Table 3 ) and 36 or 37 for test forms in group 3. The total number of questions (including questions in the domains of reading and science) was, in all cases, between 50 (form 11) and 63 (form 8). This number was determined so that the vast majority of students would be able to complete the test within the 2 h allocated (the number varies because some test questions, particularly in reading and science, were longer to solve than others, an aspect that was not necessarily related to their level of difficulty). Indeed, on average across all students, only 1.5% of items were left unanswered at the end of the test (OECD, 2014a , p.236). The content of the mathematics section is described in greater detail below (study 1) and in Supplementary Material .

In study 1, the mathematics section of the practice test (paper-based) and the criterial test (computer-based) shared most characteristics; they were developed by the same experts and according to the same framework. The item pool for both tests ensured a balanced representation of all framework aspects (e.g., mathematics content domains, processes, and response formats), with no major imbalances between the two tests. Both tests required interleaved practice of a wide range of mathematical tasks with different kinds of problems presented in an intermixed order (Rohrer, Dedrick & Stershic, 2015 ). Test developers also paid particular attention to ensuring that, to the extent possible, each of the distinct test forms administered to students did not deviate strongly in these dimensions from the characteristics of the overall item pool (see Supplementary Material for an overview of item types in each test).

No items administered in the practice test were used in the criterial test. The item sets consisted of distinct problems, even though they called on the same type of mathematics knowledge and procedures as tasks in the practice test. For example, simple proportional reasoning—an aspect of the content area “Change and relationships”—was required to solve both task DRIP RATE (question 13), a task included in the practice test, and CD PRODUCTION (question 12), a task in the criterial test (see Supplementary Material ). Similarly, in both tests, students were presented with tasks where they needed to read values from a two-dimensional chart (see CHARTS, question 7, and BODY MASS INDEX, question 17). While the scenario and content of the tasks was always different between the tasks in the criterial test and those in the practice test, all major content areas (space and shape; change and relationships; quantity; uncertainty and data) were represented in the criterial test and in each test form used in the practice session (by a minimum of two questions for test forms in group 1, of five questions for test forms in group 2, and seven or more questions for test forms in group 3). In other words, for each task in the criterial test there was always at least one task in the practice test from the same mathematics sub-domain.

In summary, the practice test and the criterial test covered the same content areas and used similar question and response formats. The main difference was the mode of administration (pen-and-paper vs. computer). Both tests consisted entirely of word problems including graphical displays of data and mathematics formulae, with the only format difference between the two being that interactive graphical displays were only included in the criterial test. Furthermore, both tests were targeted at a similar broad range of proficiencies, although lower levels of proficiency were somewhat less well covered by the criterial test.

In study 2, the practice test (T1) was identical to study 1, whereas the criterial test (T2-ps) differed from both T1 and from the criterial test used in study 1 (T2-m). The criterial test in study 2 consisted of problem-solving tasks that did not require expert knowledge to solve. The PISA 2012 assessment of problem solving focused on students’ general reasoning skills, their ability to regulate problem-solving processes, and their willingness to do so, by confronting students with “complex problems” and “analytical problems” that did not require subject-specific knowledge (e.g., of mathematics) to solve (in “complex problems,” students needed to explore the problem situation to uncover additional information required to solve the problem; this interactive knowledge acquisition was not required in “analytical problems”) (OECD, 2013 ; Ramalingam, Philpot & McCrae, 2017 ). The problem-solving competencies assessed by T2-ps are relevant to all domains (mathematics, reading, and science) assessed in T1, but the ability to solve such problems does not rely on the content and procedural knowledge assessed in any of the three domains present in T1. Example tasks from the assessment of problem solving (T2-ps) can be found in OECD ( 2014b , pp. 35–44)

All variables used in the analysis (with one exception highlighted below) are included in the public-use files for PISA 2012, and described in detail in the PISA technical report (OECD, 2014). In particular, we used the following variables in our analyses:

PISA test scores, which are included in PISA data sets as multiply imputed measures of proficiency (“plausible values”). PISA test scores are based on item-response-theory scaling procedures and are comparable across students taking different test forms. PISA scores are reported on a standardized scale, with mean 500 and standard deviation 100 across OECD member countries. Therefore, a difference of one PISA point corresponds to an effect size (Cohen’s d ) of 1% of a standard deviation, irrespective of the testing domain.

Individual and school-level socio-economic status (the PISA index of economic, social, and cultural status ), which is based on student responses to the background questionnaire.

Age (in months), sex, and grade, which are based on administrative information collected during sampling operations.

A binary indicator for students’ levels of mathematics anxiety, derived from student answers to the background questionnaire. The anxiety indicator is based on the PISA index of mathematics anxiety, which is standardized to have mean zero across OECD countries. We derive an indicator from this index that distinguishes students with positive values of math anxiety (meaning that their level of anxiety is above average) from the remaining students. Footnote 5

The only information used in this study that is not available in the publicly documented PISA files is the time lag between the pen-and-paper session (T1) and the computer-based session (T2). We contacted national centers to obtain session report forms and used these to identify the typical time lag between T1 and T2. Our analysis revealed that most national centers organized the PISA administration so that T1 would take place in the morning and T2 in the afternoon of the same day. However, in Brazil and the Slovak Republic, T2 was administered the day after T1; in Italy, T2 was conducted within a week of T1; and in Macao (China), T2 was administered 3 weeks after T1.

Overview of Analysis

Study 1 and study 2 were both analyzed according to parallel analysis plans.

In order to identify the impact of mathematical-problem-solving practice (the amount of mathematics questions in T1) on subsequent performance in the target task T2-m (study 1) or T2-ps (study 2), we conducted mean comparisons across the three treatment arms within a linear regression framework, to allow for the inclusion of control variables. The treatment consisted of exposing students to a greater dose of problem-solving practice in mathematics at the expense of problem-solving practice in other subjects (problem sets containing reading and science material). Our preferred model measures the treatment as a continuous time variable (the time-equivalent number of mathematics tasks included in the practice, which varies from 0.5 h for G1 to 1 h for G2 and 1.5 h for G2): this assumes that the dose–response relationship is (locally) linear. To identify possible non-linearities, we also compared G2 (1 h) and G3 (1.5 h) to G1 without imposing a functional form on the relationship, by including indicator variables in the regression.

Formally, let ( $ {\mathrm{y}}_{\mathrm{i}}^{\mathrm{T}2} $ ) indicate students’ performance in T2 and ( $ {\mathrm{m}}_{\mathrm{i}}^{\mathrm{T}1} $ ) indicate a “treatment” variable, measuring the intended amount of mathematics tasks in T1. Further, let α c represent a set of country-specific intercepts and the vector x i additional baseline controls. We estimated

In the above equation, error terms ϵ i are assumed to be correlated across students attending the same school and are allowed to be correlated within countries as well, but they are assumed to be independent across countries. We took the multi-level, stratified sample design into account by using balanced repeated replication weights, in line with recommended procedures for secondary data analysis in PISA (OECD, 2009 ). In our preferred specification, the “treatment” variable is a scalar variable measuring the intended amount of mathematics tasks in T1 ( $ {\mathrm{m}}_{\mathrm{i}}^{\mathrm{T}1} $ ) in hours; we also ran regressions where $ {\mathrm{m}}_{\mathrm{i}}^{\mathrm{T}1} $ represents a 2 × 1 vector with two dummy indicators for group 2 (1 h) and group 3 (1.5 h).

In study 1, we used the same regression framework to compare the demographic and socio-economic characteristics of students in the three groups and to verify the balanced nature of the three experimental groups; in this case, we excluded control variables from the regression.

To identify the moderating effect of baseline student characteristics and of the time lag between tests on our coefficient of interest β , we interacted the treatment variable with indicator variables (or vectors) for the typical time-lag between tests (a country-level variable), for sex, for performance levels in T1, and for levels of self-reported mathematics anxiety.

Sample Descriptives and Balancing Tests

Table 2 presents descriptive statistics for study 1 participants. Formal tests confirmed that the small associations of the assignment variable with baseline characteristics, captured by the “beta” coefficient in Table 2 , were well within the confidence intervals for random associations. Students’ performance on mathematics tasks in the practice session—a proxy for their baseline potential and for the effort exerted in the practice session—was also found to be equivalent across the three groups. In order to maximize our ability to detect small effects of testing, we therefore included controls for performance in T1 in subsequent analyses.

Effect of Math Test Practice on Math Performance

We investigated the effects of mathematics problem-solving practice by looking at how students’ results in the target test varied depending on their exposure to mathematics problems in the practice session. Results showed that the greater the proportion of mathematics problems included in the practice session, the better students performed on the target test. In our preferred model, which assumes a linear dose–response relationship, 1 h of additional problem-solving practice in mathematics improved students’ ability to solve mathematics problems by about 2% of a standard deviation ( b = 2.29, SE = 0.84, p = 0.007) (see model 1 in Table 3 ). Mean comparisons that do not assume a functional form for the dose–response relationship confirmed that both G2 and G3 outperformed G1 on the target task ( b G2-G1 = 2.40, SE G2-G1 = 1.09; b G3-G1 = 2.41, SE G3-G1 = 0.87) (model 2 in Table 3 ). Both differences were significant, and we could not reject the hypothesis of a linear dose–response relationship at conventional levels of significance ( p = 0.178).

Heterogeneity in Treatment Effects

We investigated the moderating influence of the time lag between the practice test and the target test and of student baseline characteristics on the effects on math performance highlighted in Table 4 . All moderation analyses are presented with mathematics performance in T1 as a control variable. Table 4 summarizes the results of moderation analyses.

Effects were positive and significant both for countries in which T1 and T2 were given on the same day ( b lag0 = 1.74, SE lag0 = 0.88) and for the three countries in which T2 was administered on a different day, but within a week of T1 ( b lag1 = 7.87, SE lag1 = 3.58) (model 7 in Table 4 ). While the latter estimate is substantially larger than the former, the large standard errors around these estimates mean that we could not exclude, at conventional levels of significance, that results were equally strong for the two sets of countries ( p = 0.103). These results clearly indicate that the observed effects were not limited to the hours immediately following the intervention and may have had a longer-term impact on student learning.

The observed learning effect of mathematics problem-solving practice was significantly larger for boys ( b boy = 4.12, SE boy = 1.41) than for girls, whose estimated effect is close to zero ( b girl = 0.34, SE girl = 1.11). A formal test rejected equal effects ( b boy – b girl = 3.78, SE = 1.89, p = 0.045) (see model 8 in Table 4 ).

Results presented in Table 4 (model 9) suggest that students who reported above-average levels of math anxiety may have benefited just as much as students with lower levels of math anxiety from test practice. Point estimates were similar for the two groups (none of which reaches statistical significance alone, due, also, to the smaller sample size for this analysis).

Mathematics Proficiency in the Practice Test

Our results were also relatively inconclusive regarding the moderating effect of mathematics achievement. For the purpose of this moderation analysis, students were assigned to one of five proficiency groups depending on their score in mathematics in T1; levels 2, 3, and 4 are the same as the corresponding levels of mathematics literacy identified and described by the mathematics expert group that guided the development of the assessment (OECD, 2014a , p. 297). Level 1 includes all students whose performance lay below the lower score limit for level 2; level 5 includes all student whose performance lay above the upper score limit for level 4. While the point estimates seemed to vary across the performance distribution and were individually significant only for moderate-to-high levels of prior performance (levels 3 and 4), we could not reject uniform effects across the performance distribution, meaning that the observed variation is compatible with the null hypothesis of no variation (model 10 in Table 4 ). The pattern of statistical significance may reflect the relatively poor coverage of the lower levels of proficiency in the criterial test, which made it hard for low-achieving students to demonstrate any progress (see the section on Materials above and the Supplementary Material ).

Effect of Math Test Practice on Domain-General Problem Solving

Participants in study 2 were assessed on a criterial test not including any mathematics questions. We found no difference in performance among the 19,481 students who sat a test aimed at measuring general reasoning and problem-solving ability in T2 ( b = 0.95, SE = 1.16, p = 0.413) between groups that differed in the amount of mathematics test practice (see models 5 and 6 in Table 3 ).

Our results shed new light on the role of tests in education. Teachers generally use tests as a way to evaluate the skills students have acquired, to certify students’ knowledge, or to adapt their teaching to the pace of progress of heterogeneous student populations. They tend to consider tests as assessment tools rather than learning tools (Wooldridge et al., 2014 ) . Learning is usually regarded as the product of information-encoding activities, such as lectures, seminars, and independent study. Tests, on the other hand, are considered to be, at best, neutral events for learning. In fact, concerns over standardized tests are mounting in many countries amidst fears that such tests reduce the time available for encoding activities, and that teachers and students will focus their efforts on what tests measure at the expense of acquiring other valuable knowledge and skills.

The experience of students participating in PISA 2012, an assessment developed to measure students’ ability to apply knowledge to real-life problems and situations, demonstrates that even in the absence of corrective feedback, participation in a one-session low-stakes test is not associated with lower performance and, in fact, with a small positive gain in mathematical problem-solving skills. Importantly, the learning effects that we estimate appear to persist over medium time lags (1–7 days).

Crucially, because our design compared students sitting tests that differed only in the amount of content-relevant and content-irrelevant material, our estimates refer to benefits that come on top of improvements in generic test-taking skills that students may acquire by completing a practice test.

Our results indicate that one additional hour spent solving test questions that require students to practice the application of subject-specific principles and procedures was associated with an improved performance of 2% of a SD in students’ ability to retrieve and use such principles and procedures to solve new sets of problems. While the effect may appear to be very small according to standard levels first introduced by Cohen ( 1988 ), Funder and Ozer ( 2019 ) note that when reliably estimated (a condition that our analysis satisfies), what is typically considered to be a very small effect for the explanation of single events can have potentially consequential effects. Moreover, effect sizes need to be evaluated and classified within a frame of reference. In our context, if test practice effects are additive (a condition not rejected by our data), spending longer or repeated test sessions may be associated with larger effects. In fact, evidence from the literature indicates that the effect of retrieval practice is greater when such practice takes place in multiple sessions (Roediger & Butler, 2011 ). Assuming a linear dose–response relationship, our results suggest that five additional hours spaced across time may be associated with an improved performance of about 10% of a SD. This effect is comparable to the estimated average difference in mathematics performance between males and females at age 15 (OECD, 2015 ).

Our findings therefore indicate that by practicing problem solving in mathematics, students learn, first and foremost, how to solve mathematics problems and, in line with our hypothesis, that the testing effect depends on the amount of matching effort expended during retrieval practice.

We found that test-practice effects were positive on average, but also that test practice might have widened existing sex differences in mathematics performance because its benefits were larger for boys. This finding should be considered and evaluated given existing gender gaps in mathematics proficiency, particularly among high-achieving students (Hedges & Nowell, 1995 ; OECD, 2015 ). Contrary to our hypothesis, we did not find differences across students with different levels of mathematics anxiety. This result may be due to the fact that the PISA test is low-stakes, meaning that neither performance on the practice test, nor performance on the criterial test, had any consequence for test takers. The absence of a moderating effect for mathematics anxiety may not be generalizable to situations in which tests are consequential for respondents.

Do the performance differences observed in the target task imply that students’ mathematics skills improved? While prior studies, with different settings and on different populations, found testing effects similar to those described in study 1 (see for example Adesope, Trevisan, & Sundararajan, 2017 and Pan & Rickard, 2018 for comprehensive reviews), theories of transfer-appropriate processing cannot exclude that the testing effect observed in those studies occurred because students gained test-taking abilities and transferred those from practice tests to target tests, rather than the domain-specific principles and procedures which educators care most about. We developed two studies to test the hypothesis that the testing effect is driven by the amount of matching effort expended during retrieval practice and exclude that students merely became better at solving tests, rather than at mathematics, unlike most studies of testing effects. In both studies, all participants, irrespective of the specific group they were randomly assigned to, sat a 2-h test during the practice session. In study 1, we induced variation in the amount of matching effort expended during retrieval practice by administering a different amount of mathematics questions to different groups of students. Participants in the “control” group, G1, Footnote 6 spent less time than the treatment group answering mathematics problems in the 2-h test and dedicated this time to answering a combination of text comprehension and science problems. G1 therefore received a “placebo” treatment that could equally well have taught students general problem-solving and test-taking skills (the amount of effort was the same but such effort was directed at material that did not match the content present in the criterial test). In study 2, we compared the performance of groups that differed in the amount of mathematics test practice on a criterial test that did not include any mathematics questions, but rather questions directed at assessing general problem-solving abilities. We found no significant difference. The learning effects highlighted in study 1 must therefore be interpreted as the effect of solving mathematics problems on mathematics problem-solving performance, above and beyond any effect of test practice on test performance in general.

The literature on testing effects suggests that providing corrective feedback and offering multiple short test-practice sessions over time can significantly amplify the effect of test taking on subsequent performance (McDermott et al., 2014 ; Roediger & Butler, 2011 ). Moreover, feedback by teachers can negate stereotype threat affecting girls’ performance in mathematics (Cohen, Garcia, Apfel, & Master, 2006 ).

Our study suggests that lessons that include targeted test-practice sessions bear the promise of improving students’ ability to transfer their knowledge of principles and procedures to new, real-world problems, although effects are not large and appear to accrue, in the absence of feedback, only to boys. Future research should aim to identify and compare how the effect of the type of retrieval practice that we studied is associated with intensity and spacing, overall and among key population subgroups. For example, it is important to identify what is the learning gain that is associated, for example, with 10 h of test practice administered in five sessions of 2 h each or in ten sessions of 1 h each. Future research on retrieval practice should also systematically explore differential effects between boys and girls (and men and women), to establish if the differences highlighted in this study can be generalized to other settings. It would be equally important to establish if changes in conditions, for example, the provision of feedback, might ensure that girls benefited from test practice as much as boys. Finally, our study examined test practice in mathematics and it would be important to consider if findings are generalizable to how students acquire proficiency in mastering principles and procedures that are necessary to solve problems in other domains.

We use the word “country” to refer to all national and subnational entities participating in PISA as distinct territories. In most cases, they correspond to entire countries, although in some circumstances they refer to sub-national entities or economies. While 65 national samples exist for PISA 2012, only 32 (out of 65) countries that participated in the PISA pen-and-paper assessment opted for conducting a computer-based assessment of students’ mathematical skills. Countries may have decided not to participate in the optional computer-based assessment because of both the direct costs (participation in optional assessments determines additional international development costs as well as national implementation costs for countries) and indirect costs (increased administrative burden of the PISA administration) involved. Moreover, some countries did not feel that they possessed the required technical capacity to administer a computer-based assessment in 2012.

Students are included in the PISA sample if they participate in at least one cognitive test session or if they complete a significant fraction of a background questionnaire.

We define attendance in a test as having at least one non-missing response to an assessment task in that test.

General information about the content of PISA tests, such as the test framework and sample tasks from previous PISA assessments, are shared in advance of the test with participating schools and are also available on the web, to ensure informed consent of schools (and depending on national practice and legislation, of students’ families) to participate. To our knowledge, students do not prepare specifically for the test. Test frameworks define vast domains of knowledge and typically refer to curricular contents that students have learned over several years of study.

Because the questions about mathematics anxiety are included in a rotated part of the student questionnaire, the anxiety indicator is available for only about two thirds of the sample. Apart from a small amount of item non-response, missing information about mathematics anxiety is “missing completely at random” due to the randomized assignment of blocks of questions in the student background questionnaire.

We refer to the treatment groups as G1, G2, and G3 in both study 1 and study 2, to underscore that these groups received the same “treatment” (the same practice test) regardless of the study. As explained above, however, study 1 and study 2 consist of different participants.

Abbott, E. E. (1909). On the analysis of the factors of recall in the learning process. Psychological Monographs, 11 , 159–177.

Google Scholar

Adesope, O. O., Trevisan, D. A., & Sundararajan, N. (2017). Rethinking the use of tests: A meta-analysis of practice testing. Review of Educational Research, 87 (3), 659–701.

Agarwal, P. K., Karpicke, J. D., Kang, S. H. K., Roediger, H. L., & McDermott, K. B. (2008). Examining the testing effect with open- and closed-book tests. Applied Cognitive Psychology, 22 (7), 861–876. https://doi.org/10.1002/acp.1391 .

Article Google Scholar

Amrein, A. L., & Berliner, D. C. (2002). High-stakes testing, uncertainty, and student learning. Education Policy Analysis Archives, 10 (18). Retrieved on October 25 at http://epaa.asu.edu/epaa/v10n18/

Anderson, M. C., Bjork, R. A., & Bjork, E. L. (1994). Remembering can cause forgetting: Retrieval dynamics in long-term memory. Journal of Experimental Psychology: Learning, Memory, and Cognition, 20 (5), 1063–1087. https://doi.org/10.1037/0278-7393.20.5.1063 .

Ashcraft, M. H., & Kirk, E. P. (2001). The relationships among working memory, math anxiety, and performance. Journal of Experimental Psychology: General, 130 (2), 224–237. https://doi.org/10.1037/0096-3445.130.2.224 .

Ashcraft, M. H., & Ridley, K. S. (2005). Math anxiety and its cognitive consequences: A tutorial review. In Handbook of mathematical cognition (pp. 315–327). New York: Psychology Press.

Au, W. (2007). High-stakes testing and curricular control: A qualitative metasynthesis. Educational Researcher, 36 (5), 258–267.

Barnett, S. M., & Ceci, S. J. (2002). When and where do we apply what we learn? A taxonomy for far transfer. Psychological Bulletin, 128 (4), 612–637.

Beilock, S. L., & Carr, T. H. (2005). When high-powered people fail: Working memory and “choking under pressure” in math. Psychological Science, 16 (2), 101–105.

Bietenbeck, J. (2014). Teaching practices and cognitive skills. Labour Economics, 30 , 143–153.

Bjork, R. A. (1994). Memory and metamemory considerations in the training of human beings. In J. Metcalfe & A. Shimamura (Eds.), Metacognition: Knowing about knowing (pp. 185–205). Cambridge: MIT Press.

Bjork, R. A. (1999). Assessing our own competence: Heuristics and illusions. In D. Gopher & A. Koriat (Eds.), Attention and performance XVII: Cognitive regulation of performance: Interaction of theory and application (pp. 435–459). Cambridge: MIT Press.

Bjork, R.A., & Bjork, E.L. (1992). A new theory of disuse and an old theory of stimulus fluctuation. In A. Healy, S. Kosslyn, & R. Shiffrin (Eds.), From learning processes to cognitive processes: Essays in honor of William K. Estes (Vol. 2, pp. 35–67). Hillsdale NJ: Erlbaum.

Butler, A. C. (2010). Repeated testing produces superior transfer of learning relative to repeated studying. Journal of Experimental Psychology: Learning, Memory, and Cognition, 36 (5), 1118–1133. https://doi.org/10.1037/a0019902 .

Butler, A. C., & Roediger, J. L. (2008). Feedback enhances the positive effects and reduces the negative effects of multiple-choice testing. Memory and Cognition, 36 (3), 604–616.

Carpenter, S. K. (2009). Cue strength as a moderator of the testing effect: The benefits of elaborative retrieval. Journal of Experimental Psychology. Learning, Memory, and Cognition, 35 (6), 1563–1569.

Carpenter, S. K. (2012). Testing enhances the transfer of learning. Current Directions in Psychological Science, 21 (5), 279–283.

Carpenter, S. K., & Delosh, E. L. (2006). Impoverished cue support enhances subsequent retention: Support for the elaborative retrieval explanation of the testing effect. Memory & Cognition, 34 (2), 268–276. https://doi.org/10.3758/BF03193405 .

Carrier, M., & Pashler, H. (1992). The influence of retrieval on retention. Memory & Cognition, 20 (6), 633–642. https://doi.org/10.3758/BF03202713 .

Coburn, C. E., Hill, H. C., & Spillane, J. P. (2016). Alignment and accountability in policy design and implementation: The common core state standards and implementation research. Educational Researcher, 45 , 243–251. https://doi.org/10.3102/0013189X16651080 .

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates, Publishers.

Cohen, G. L., Garcia, J., Apfel, N., & Master, A. (2006). Reducing the racial achievement gap: A social-psychological intervention. Science, 313 (5791), 1307–1310.

Crocco, M. S., & Costigan, A. T. (2007). The narrowing of curriculum and pedagogy in the age of accountability: Urban educators speak out. Urban Education, 42 (6), 512–535.

Dirkx, K. J. H., Kester, L., & Kirschner, P. A. (2014). The testing effect for learning principles and procedures from texts. The Journal of Educational Research, 107 (5), 357–364. https://doi.org/10.1080/00220671.2013.823370 .

Entwistle, N., & McCune, V. (2004). The conceptual bases of study strategy inventories. Educational Psychology Review, 16 (4), 325–345.

Fazio, L. K., Agarwal, P. K., Marsh, E. J., & Roediger, H. L. (2010). Memorial consequences of multiple-choice testing on immediate and delayed tests. Memory & Cognition, 38 (4), 407–418.

Foley, A. E., Herts, J. B., Borgonovi, F., Guerriero, S., Levine, S. C., & Beilock, S. L. (2017). The math anxiety-performance link: A global phenomenon. Current Directions in Psychological Science, 26 (1), 52–58.

Funder, D. C., & Ozer, D. J. (2019). Evaluating effect size in psychological research: Sense and nonsense. Advances in Methods and Practices in Psychological Science, 2 (2), 156–168. https://doi.org/10.1177/2515245919847202 .

Hedges, L. V., & Nowell, A. (1995). Sex differences in mental test scores, variability, and numbers of high-scoring individuals. Science, 269 (5220), 41–45.

Kang, S. H. K., McDermott, K. B., & Roediger, H. L. (2007). Test format and corrective feedback modify the effect of testing on long-term retention. European Journal of Cognitive Psychology, 19 (4–5), 528–558. https://doi.org/10.1080/09541440601056620 .

Karpicke, J. D. (2012). Retrieval-based learning active retrieval promotes meaningful learning. Current Directions in Psychological Science, 21 (3), 157–163. https://doi.org/10.1177/0963721412443552 .

Karpicke, J. D., & Blunt, J. R. (2011). Retrieval practice produces more learning than elaborative studying with concept mapping. Science, 331 (6018), 772–775. https://doi.org/10.1126/science.1199327 .

Karpicke, J. D., & Roediger, H. L. (2008). The critical importance of retrieval for learning. Science, 319 (5865), 966–968. https://doi.org/10.1126/science.1152408 .

Keresztes, A., Kaiser, D., Kovács, G., & Racsmány, M. (2014). Testing promotes long-term learning via stabilizing activation patterns in a large network of brain areas. Cerebral Cortex, 24 (11), 3025–3035. https://doi.org/10.1093/cercor/bht158 .

Lyle, K. B., Bego, C. R., Hopkins, R. F., Hieb, J. L., & Ralston, P. A. S. (2019). How the amount and spacing of retrieval practice affect the short- and long-term retention of mathematics knowledge. Educational Psychology Review , 1–19. https://doi.org/10.1007/s10648-019-09489-x .

Lyons, I. M., & Beilock, S. L. (2011). Mathematics anxiety: Separating the math from the anxiety. Cerebral Cortex, 22 (9), 2102–2110.

Lyons, I. M., & Beilock, S. L. (2012). When math hurts: Math anxiety predicts pain network activation in anticipation of doing math. PLoS One, 7 (10), e48076.

Ma, X., & Johnson, W. (2008). Mathematics as the critical filter: Curricular effects on gendered career choices. In H. M. G. Watt & J. S. Eccles (Eds.), Gender and occupational outcomes: Longitudinal assessments of individual, social, and cultural influences (pp. 55–83). Washington, DC, US: American Psychological Association.

Madaus, G. F. (1988). The influence of testing on the curriculum. In L. N. Tanner (Ed.), Critical issues in curriculum: Eighty-seventh yearbook of the National Society for the Study of Education (pp. 83–121). Chicago: University of Chicago Press.

Marsh, E. J., Roediger, H. L., Bjork, R. A., & Bjork, E. L. (2007). The memorial consequences of multiple-choice testing. Psychonomic Bulletin & Review, 6 , 194–199.

McDaniel, M. A., & Fisher, R. P. (1991). Tests and test feedback as learning sources. Contemporary Educational Psychology, 16 (2), 192–201. https://doi.org/10.1016/0361-476X(91)90037-L .

McDermott, K. B., Agarwal, P. K., D’Antonio, L., Roediger, H. L., & McDaniel, M. A. (2014). Both multiple-choice and short-answer quizzes enhance later exam performance in middle and high school classes. Journal of Experimental Psychology: Applied, 20 (1), 3–21. https://doi.org/10.1037/xap0000004 .

McNeil, L. M. (2000). Contradictions of school reform: Educational costs of standardized testing . New York: Routledge.

Miyake, A., & Shah, P. (1999). Models of working memory: Mechanisms of active maintenance and executive control . Cambridge University Press.

Morris, C. D., Bransford, J. D., & Franks, J. J. (1977). Levels of processing versus transfer appropriate processing. Journal of Verbal Learning and Verbal Behavior, 16 , 519–533.

Nelson, H. (2013). Testing more, teaching less: What America’s obsession with student testing costs in money and lost instructional time . New York: American Federation of Teachers.

Nichols, S. L., & Berliner, D. C. (2007). Collateral damage: How high stakes testing corrupts America’s schools . Cambridge, MA: Harvard Education Press.

OECD. (2009). PISA Data Analysis Manual: SPSS (2nd ed.). Paris: OECD Publishing.

OECD. (2010). Mathematics teaching and learning strategies in PISA . Paris: OECD Publishing.

OECD. (2013). PISA 2012 assessment and analytical framework: Mathematics, reading, science, problem solving and financial literacy . Paris: OECD Publishing.

OECD. (2014a). PISA technical report . Paris: OECD Publishing.

OECD. (2014b). PISA 2012 results: Creative problem solving (volume V): Students’ skills in tackling real-life problems . Paris: OECD Publishing.

OECD. (2015). The ABC of gender equality in education: Aptitude, behaviour, confidence . Paris: OECD Publishing.

Pan, S. C., & Rickard, T. C. (2018). Transfer of test-enhanced learning: Meta-analytic review and synthesis. Psychological Bulletin, 144 (7), 710–756.

Park, D., Ramirez, G., & Beilock, S. L. (2014). The role of expressive writing in math anxiety. Journal of Experimental Psychology: Applied, 20 (2), 103–111.

Pastötter, B., & Bäuml, K.-H. T. (2014). Retrieval practice enhances new learning: The forward effect of testing. Frontiers in Psychology, 5 , 286. https://doi.org/10.3389/fpsyg.2014.00286 .

Pritchard, A. (2013). Ways of learning: Learning theories and learning styles in the classroom . Routledge.

Pyc, M. A., & Rawson, K. A. (2009). Testing the retrieval effort hypothesis: Does greater difficulty correctly recalling information lead to higher levels of memory? Journal of Memory and Language, 60 (4), 437–447. https://doi.org/10.1016/j.jml.2009.01.004 .

Raghubar, K. P., Barnes, M. A., & Hecht, S. A. (2010). Working memory and mathematics: A review of developmental, individual difference, and cognitive approaches. Learning and Individual Differences, 20 (2), 110–122.

Ramalingam, D., Philpot, R., & Mccrae, B. (2017). The PISA 2012 assessment of problem solving. In B. Csapó & J. Funke (Eds.), The nature of problem solving: Using research to inspire 21st century learning . Paris: OECD Publishing.

Rawson, K. A., Vaughn, K. E., & Carpenter, S. K. (2014). Does the benefit of testing depend on lag, and if so, why? Evaluating the elaborative retrieval hypothesis. Memory & Cognition, 43 (4), 619–633. https://doi.org/10.3758/s13421-014-0477-z .

Roediger, H. L., Agarwal, P. K., Kang, S. H. K., & Marsh, E. J. (2010). Benefits of testing memory: Best practices and boundary conditions. In G. M. Davies & D. B. Wright (Eds.), New frontiers in applied memory . Brighton: Psychology Press.

Roediger, H. L., & Butler, A. C. (2011). The critical role of retrieval practice in long-term retention. Trends in Cognitive Sciences, 15 (1), 20–27.

Roediger, H. L., & Karpicke, J. D. (2006). Test-enhanced learning taking memory tests improves long-term retention. Psychological Science, 17 (3), 249–255. https://doi.org/10.1111/j.1467-9280.2006.01693.x .

Rohrer, D., Taylor, K., & Sholar, B. (2010). Tests enhance the transfer of learning. Journal of Experimental Psychology. Learning, Memory, and Cognition, 36 (1), 233–239.

Rohrer, D., Dedrick, R. F., & Stershic, S. (2015). Interleaved practice improves mathematics learning. Journal of Educational Psychology, 107 , 900–908.

Rothman, R. (2011). Something in common: The common core standards and the next chapter in American education . Cambridge, MA: Harvard Education Press.

Rubin, J. (1981). Study of cognitive processes in second language learning. Applied Linguistics, 2 , 117.

Salomon, G., & Perkins, D. N. (1989). Rocky roads to transfer: Rethinking mechanisms of a neglected phenomenon. Educational Psychology, 24 , 113–142.

Smith, M. L. (1991). Put to the test: The effects of external testing on teachers. Educational Researcher, 20 (5), 8–11.

Soderstrom, N. C., & Bjork, R. A. (2015). Learning versus performance: An integrative review. Perspectives on Psychological Science, 10 (2), 176–199.

Storm, B. C., & Levy, B. J. (2012). A progress report on the inhibitory account of retrieval-induced forgetting. Memory & Cognition, 40 (6), 827–843. https://doi.org/10.3758/s13421-012-0211-7 .

Szpunar, K. K., McDermott, K. B., & Roediger, H. L. (2008). Testing during study insulates against the buildup of proactive interference. Journal of Experimental Psychology: Learning, Memory, and Cognition, 34 (6), 1392–1399. https://doi.org/10.1037/a0013082 .

Thomas, A. K., & McDaniel, M. A. (2007). The negative cascade of incongruent generative study-test processing in memory and metacomprehension. Memory & Cognition, 35 (4), 668–678.

Thompson, C. P., Wenger, S. K., & Bartling, C. A. (1978). How recall facilitates subsequent recall: A reappraisal. Journal of Experimental Psychology: Human Learning and Memory, 4 , 210–221.

Tienken, C. H., & Zhao, Y. (2010). Common core national curriculum standards: More questions and answers. AASA Journal of Scholarship and Practice, 6 (3), 3–10.

Toppino, T. C., & Ann Brochin, H. (1989). Learning from tests: The case of true–false examinations. The Journal of Educational Research, 83 (2), 119–124.

Toppino, T. C., & Cohen, M. S. (2009). The testing effect and the retention interval. Experimental Psychology, 56 (4), 252–257. https://doi.org/10.1027/1618-3169.56.4.252 .

Toppino, T. C., & Luipersbeck, S. M. (1993). Generality of the negative suggestion effect in objective tests. The Journal of Educational Research, 86 (6), 357–362.

Vinovskis, M. (2008). From a nation at risk to no child left behind: National education goals and the creation of federal education policy . New York, NY: Teachers College Press.

Watanabe, M. (2007). Displaced teacher and state priorities in a high stakes accountability context. Educational Policy, 21 (2), 311–368.

Weinstein, C. E., Ridley, D. S., Dahl, T., & Weber, E. S. (1989). Helping students develop strategies for effective learning. Educational Leadership, 46 (4), 17–19.

Wheeler, M., Ewers, M., & Buonanno, J. (2003). Different rates of forgetting following study versus test trials. Memory, 11 (6), 571–580. https://doi.org/10.1080/09658210244000414 .

Wooldridge, C. L., Bugg, J. M., McDaniel, M. A., & Liu, Y. (2014). The testing effect with authentic educational materials: A cautionary note. Journal of Applied Research in Memory and Cognition, 3 (3), 214–221. https://doi.org/10.1016/j.jarmac.2014.07.001 .

Young, C. B., Wu, S. S., & Menon, V. (2012). The neurodevelopmental basis of math anxiety. Psychological Science, 23 (5), 492–501.

Download references

Acknowledgments

The authors would like to thank participants in seminars at the Organisation for Economic Co-operation and Development, the Paris School of Economics, Education Testing Service and University College London. They would also like to thank the editor, Fred Paas, several anonymous referees as well as the following individuals for providing input, comments and feedback on previous versions of the manuscript: Sola Adesope, Andrew Elliot, Samuel Greiff, Keith Lyle, John Jerrim, Roberto Ricci, Richard Roberts, Henry Roediger, Matthias von Davier, Allan Wigfield, Kentaro Yamamoto. The manuscript was much improved thanks to them and any errors remain our own. F.B. acknowledges support from the British Academy’s Global Professorship programme.

Author information

Authors and affiliations.

Directorate for Education and Skills, Organisation for Economic Co-operation and Development, 2 Rue André Pascal, 75016, Paris, France

Francesco Avvisati & Francesca Borgonovi

Department of Social Science, Institute of Education, University College London, 55-59 Gordon Square, London, WC1H 0NU, United Kingdom

Francesca Borgonovi

You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Francesca Borgonovi .

Ethics declarations

Conflict of interest.

F.A. and F.B. work at the Organisation for Economic Co-operation and Development (OECD), the organization responsible for conducting the PISA study. The opinions expressed and arguments employed in this article are those of the authors and do not necessarily represent the official view of the OECD, its member countries or the British Academy.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

(DOCX 1925 kb)

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Avvisati, F., Borgonovi, F. Learning Mathematics Problem Solving through Test Practice: a Randomized Field Experiment on a Global Scale. Educ Psychol Rev 32 , 791–814 (2020). https://doi.org/10.1007/s10648-020-09520-6

Download citation

Published : 04 February 2020

Issue Date : September 2020

DOI : https://doi.org/10.1007/s10648-020-09520-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Retrieval practice
Large-scale international assessments
Mathematics
Problem solving
Find a journal
Publish with us
Track your research

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Publications
Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

Advanced Search
Journal List
Front Psychol

Potential for Assessing Dynamic Problem-Solving at the Beginning of Higher Education Studies

Benő csapó.

1 MTA-SZTE Research Group on the Development of Competencies, University of Szeged, Szeged, Hungary

Gyöngyvér Molnár

2 Department of Learning and Instruction, University of Szeged, Szeged, Hungary

There is a growing demand for assessment instruments which can be used in higher education, which cover a broader area of competencies than the traditional tests for disciplinary knowledge and domain-specific skills, and which measure students' most important general cognitive capabilities. Around the age of the transition from secondary to tertiary education, such assessments may serve several functions, including selecting the best-prepared candidates for certain fields of study. Dynamic problem-solving (DPS) is a good candidate for such a role, as tasks that assess it involve knowledge acquisition and knowledge utilization as well. The purpose of this study is to validate an online DPS test and to explore its potential for assessing students' DPS skills at the beginning of their higher education studies. Participants in the study were first-year students at a major Hungarian university ( n = 1468). They took five tests that measured knowledge from their previous studies: Hungarian language and literature, mathematics, history, science and English as a Foreign Language (EFL). A further, sixth test based on the MicroDYN approach, assessed students' DPS skills. A brief questionnaire explored learning strategies and collected data on students' background. The testing took place at the beginning of the first semester in three 2-h sessions. Problem-solving showed relatively strong correlations with mathematics ( r = 0.492) and science ( r = 0.401), and moderate correlations with EFL ( r = 0.227), history ( r = 0.192), and Hungarian ( r = 0.125). Weak but still significant correlations were found with certain learning strategies, positive correlations with elaboration strategies, and a negative correlation with memorization strategies. Significant differences were observed between male and female students; men performed significantly better in DPS than women. Results indicated the dominant role of the first phase of solving dynamic problems, as knowledge acquisition correlated more strongly with any other variable than knowledge utilization.

Introduction

The social and economic developments of the past decades have re-launched the debate on the mission of schooling, more specifically, on the types of skills schools are expected to develop in their students in order to prepare them for an unknown future. One of the most characteristic features of these debates is a search for a new conception of the knowledge and skills students are expected to master (see e.g., Adey et al., 2007 ; Binkley et al., 2012 ; Greiff et al., 2014 ). These developments and expectations have reached higher education as well, and novel assessment needs have emerged to reflect the changes. Tests in higher education have traditionally been used as a part of the selection processes (entrance examinations) and to assess students' level of mastery, mostly in the form of summative tests based on the disciplinary content of courses. Recently, the functions of assessments have significantly expanded, thus requiring a renewal of assessment processes in a number of dimensions.

This study lies at the intersection of three rapidly developing fields of research on higher education. The context of the research is set by the practical needs of (1) developing new assessment methods for higher education, including innovative and efficient selection processes for choosing students for higher education studies, and assessing university outcomes beyond disciplinary knowledge and domain-specific skills. These demands have directed the attention of researchers to (2) the twenty-first-century skills as desired outcomes of higher education. Rapidly developing technology-based assessment has made it possible to measure several twenty-first-century skills and to include them in large-scale assessments. (3) Dynamic problem-solving (DPS) is one of those skills which by now has an established research background and may satisfy the needs of higher education. Solving problems in the process of being assessed in DPS based on computer-simulated scenarios involves the component skills of scientific reasoning, knowledge acquisition and knowledge utilization, all necessary for successful higher education studies (see e.g., Buchner and Funke, 1993 ; Funke, 2001 ; Greiff et al., 2012 ; Csapó and Funke, 2017a ; Funke and Greiff, 2017 ). The processes of solving problems in computer-simulated scenarios involve the component skills of scientific reasoning, knowledge acquisition and knowledge utilization, all necessary for learning effectively in higher education (see e.g., Buchner and Funke, 1993 ; Funke, 2001 ; Greiff et al., 2012 ; Csapó and Funke, 2017a ; Funke and Greiff, 2017 ).

As the main construct explored in the present study, DPS has already been defined and assessed in several previous studies. The problems are built on formal models represented by a finite-state automaton, where the output signals are determined by the input signal (Buchner and Funke, 1993 ). “In contrast to static problems, computer-simulated scenarios provide the unique opportunity to study human problem-solving and decision-making behavior when the task environment changes concurrently to subjects' actions. Subjects can manipulate a specific scenario via a number of input variables […] and they observe the system's state changes in a number of output variables. In exploring and/or controlling a system, subjects have to continuously acquire and use knowledge about the internal structure of the system” (Blech and Funke, 2005 , p. 3).

In the present study, DPS was assessed with a computerized solution based on the MicroDYN approach (Greiff and Funke, 2009 ; Funke and Greiff, 2017 ) similar to that employed in delivering most interactive items in the innovative domain in PISA 2012. That assessment framework defined problem-solving in a more general way: “Problem-solving competency is an individual's capacity to engage in cognitive processing to understand and resolve problem situations where a method of solution is not immediately obvious. It includes the willingness to engage with such situations in order to achieve one's potential as a constructive and reflective citizen.” (OECD, 2013a , p. 122). An interpretation of this definition follows: “What distinguishes the 2012 assessment of problem-solving from the 2003 assessment is not so much the definition of problem-solving competency, but the mode of delivery of the 2012 assessment (computer-based) and the inclusion of problems that cannot be solved without the solver interacting with the problem situation” (OECD, 2013a , p. 122). The PISA 2012 problem-solving assessment included both static and interactive tasks, and in this context interactivity is defined as “Interactive: not all information is disclosed; some information has to be uncovered by exploring the problem situation” (OECD, 2014 , p. 31, Fig. V.1.2). In the present study, all items are interactive, so the construct we assess is identical with the one PISA assessed in 2012 with its interactive items.

Theoretical framework

Context of the study: need for new assessments in higher education.

The need to develop new assessment instruments for higher education has emerged both at international and national levels in a number of countries. There is a general intention to adapt the content of the assessments to changed expectations of the outcomes of higher education. The altered content may then require new assessment methods (see e.g., Bryan and Clegg, 2006 ). There is a change in the purpose of assessments as well as a visible intention to introduce the principles of evidence-based decision-making and accountability processes to higher education (Hutchings et al., 2015 ; Ikenberry and Kuh, 2015 ; Zlatkin-Troitschanskaia et al., 2015 ). The new functions of assessment go beyond the usual applications of summative tests to measure the mastery level of courses and include estimating educational added value of particular phases of studies, or entire training programs. As there is a great variety of competencies that are outcomes of higher education, thus limiting inter-institutional comparisons in terms of domain-specific competencies, we see a growing need to measure and compare domain-general competencies.

These intentions are clearly marked by feasibility studies launched by the OECD to compare the achievement of college and university students in a number of countries (Assessment of Higher Education Learning Outcomes, AHELO). The AHELO program included assessment of domain-specific competencies as well as of generic cognitive skills, for which the test tasks were adapted from the Collegiate Learning Assessment instrument (Tremblay et al., 2012 ). Another international initiative, the TUNING CALOHEE project (Measuring and Comparing Achievements of Learning Outcomes in Higher Education in Europe), intends to create an assessment system to compare the outcomes of universities in Europe (Coates, 2016 ).

In the United States, as the century-long history of successfully administering the Scholastic Aptitude Test (SAT) indicates, admissions processes have always been based on assessing generic cognitive skills (Atkinson and Geiser, 2009 ). As studies show, the SAT tests predict achievement in higher education beyond the high school grade point average. They comprise mathematical and verbal components (factor analysis with a recent version of it confirmed the two-factor model, see Wiley et al., 2014 ), while university admissions in many countries have usually been based on assessing domain-specific competencies (Zlatkin-Troitschanskaia et al., 2015 ).

A closer context of the study is Hungarian higher education and the admissions process used by its institutions. As there is no specific entrance examination, admissions are based on matriculation examination results. The matriculation examination, like so many other European countries, was introduced in Hungary in the mid-nineteenth century, and it has changed relatively little during its long history. At present, there are three mandatory subjects: (1) Hungarian language and literature, (2) mathematics, and (3) history. Beyond these, students must choose a further subject out of a large number of electives. An examination can be taken at two levels in any subject; there is an intermediate and an advanced exam. There is no exact (measurable) definition for the differences between the two levels. Intermediate exams are taken at students' schools before committees formed from teachers in their own schools, while the advanced exams are centralized and are taken before (independent) committees formed from teachers in other schools. The admissions scores are computed by complex formulas; for advanced exams, extra scores are awarded, and other factors may also be taken into account.

The inadequacy of such a selection criterion is widely discussed, but few research results are available to make evidence-based judgments about the validity of the current practice and about potential alternative solutions. It seems possible that a reformed matriculation examination could serve to certify completion of secondary studies and at the same time could act as a major component of the admissions process (Csapó, 2009 ). Such a matriculation examination should measure students' knowledge at one level but on a scale which represents a broad range of achievement, should be a technology-based assessment (possibly using item banks and adaptive testing), and should include a few (probably five) compulsory subjects without electives.

The new admissions processes are expected to provide a better prediction of students' success in a changed world of higher education than those of the traditional methods introduced so many decades ago. Assessment of generic cognitive skills, possibly a representative member of the twenty-first-century skills, could also be a component of a new admissions process. To explore the feasibility and validity of such an admissions model, we have measured five domain-specific competencies plus dynamic problem-solving, and we report the results in the present study.

Definition and technology-based assessment of twenty-first-century skills in educational settings

A number of studies have analyzed the requirements of the knowledge-based economy and concluded that science, technology, engineering, and mathematics (STEM) education should be strengthened and that skills relevant to a dynamically changing technology-rich environment should be developed. In this context, societies today and in the foreseeable future are characterized by a new group of skills, which are often called the twenty-first-century skills, or, in other contexts, transversal skills (Greiff et al., 2014 ). This loosely defined set of skills includes problem-solving, information and communication skills, critical thinking, creativity, entrepreneurship and collaboration. The topic of twenty-first-century skills has become popular in the literature on the future of education (Trilling and Fadel, 2009 ; National Research Council, 2013 ; Kong et al., 2014 ), and a number of projects have been launched to define, assess, and develop these skills.

Although most skills identified under this label are not new in the sense that they have not been studied before or that they have not been relevant in everyday life, the way they are utilized in this century may be novel. The main novelty is that these skills today are mostly used in a technology-rich environment. Therefore, they should be measured by means of technology. This approach is demonstrated by the Assessment and Teaching of twenty-first-Century Skills (ATC21S) project, among other studies. The first phase of the ATC21S project dealt with definitions and psychometric, technological and policy issues (Griffin et al., 2012 ), while the second phase focused on the assessment of collaborative problem-solving (Griffin and Care, 2015 ).

Technology-based assessment has a number of advantages over traditional paper-and-pencil tests in a number of respects. Computerized tests, especially assessments delivered online, may make the entire assessment process more reliable and valid, faster, easier, and less expensive. Beyond these general benefits, there are some constructs which could not be measured without computers. There are domains where technology use is central to the definition of the domain (e.g., information-communication literacy and digital reading), while in other cases it would not be possible to implement the assessment process without technology (Csapó et al., 2012 ). DPS is such a construct, as students interact with computer-simulated systems during the testing process. Technology is the best means not only to assess these skills, but to develop them as well; for example, simulation- and game-based learning may provide an authentic learning environment to practice these skills (see Qian and Clark, 2016 ).

Those projects whose aim it was to precisely identify the twenty-first-century skills were able to define only a few of them in a measurable format (Binkley et al., 2012 ). Even fewer of those skills have an established research background that makes it possible to use them in a large-scale project. Of these, problem-solving, both dynamic (Greiff et al., 2014 ) and collaborative (OECD, 2013b ; Griffin and Care, 2015 ; Neubert et al., 2015 ), is sufficiently developed for broader practical use. Beyond these strengths, DPS is a good representative of the twenty-first-century skills because, through its component skills, it may overlap with several other complex skills in this group.

Assessment of dynamic problem-solving

Problem-solving is one of the most commonly noted constructs among the “new” twenty-first-century skills; it also has a long history in cognitive research (see Fischer et al., 2017 ). By now, cognitive research has identified a number of different types of problem-solving which can be classified by several aspects. Domain-specific problem-solving can be distinguished from the domain-general kind, analytical from complex, and static from interactive. In the present study, we deal with the assessment of dynamic problem-solving, which is interactive and can be considered as a specific form of complex problem-solving. Dynamic problem-solving, as was shown in the previous section, can only be measured by means of technology.

Complex problem-solving has already been studied in a number of contexts; previous research shows that it is a generic cognitive skill, but is different from general intelligence (Funke, 2010 ; Wüstenberg et al., 2012 ; Greiff et al., 2013a ). Using computers to assess problem-solving has allowed a migration of previous paper-based tests to an electronic platform, thus improving the efficiency and usability of the tests as well as opening up a range of new prospects (Wirth and Klieme, 2003 ). These new possibilities include constructing more real life-like scenarios, using simulations, offering interactive activities, and in this way improving the ecological validity of the assessments in general.

Using simulation to study problem-solving was already proposed long ago (Funke, 1988 ), but the broad availability of computers launched a new wave of research based on computer-simulated systems (Funke, 1993 , 1998 ; Greiff et al., 2013a ). The difficulty level of tasks based on simulation is easily scalable; even simulated minimal complex systems offer outstanding opportunities to study the processes of problem-solving (Sonnleitner et al., 2012 ; Funke, 2014 ; Funke and Greiff, 2017 ; Greiff and Funke, 2017 ).

DPS as a specific category of complex, interactive problem-solving offers outstanding potential both to create tests for laboratory studies and for large-scale assessment (Buchner and Funke, 1993 ; Funke, 2001 ; Greiff et al., 2012 ). When students solve dynamic problems on a computer, their activities can be logged and fine mechanisms of their cognition can be explored by analyzing log files (Tóth et al., 2017 ).

Several types of problem-solving have already been assessed three times within the framework of PISA. First, static problem-solving was assessed in 2003 with paper-based tests (OECD, 2004 ; Fleischer et al., 2017 ). Then, in 2012, problem-solving was measured with computerized tests comprising two types of tasks, static (15 items) and interactive (27 items). The static items were similar to those of the PISA 2003 assessment; they were computerized versions of items that would be possible to measure with paper-and-pencil tests as well, while the interactive items were novel in large-scale assessments and measured the same construct (DPS) as the present study, based on the MicroDYN approach, too (Greiff and Funke, 2009 ; Funke and Greiff, 2017 ).

The 2012 PISA assessment was the first large-scale assessment of DPS in international context and demonstrated that there were large differences between the participating countries in the problem-solving performance of their students, even if the achievement on the main literacy domains was similar (OECD, 2014 ). The successful completion of the 2012 PISA DPS assessment has accelerated research in this field and inspired a number of further studies (see Csapó and Funke, 2017a ). In PISA 2015, collaborative problem-solving was the innovative domain; collaboration was simulated by human–agent interactions (OECD, 2013b ).

Assessments of problem-solving have already proved useful in higher education, but the vast majority of them covered domain-specific problem-solving (e.g., Lopez et al., 2014 ; Zlatkin-Troitschanskaia et al., 2015 ). As technology-based assessment instruments become more widely available, such skills have been measured more often, capitalizing on experiences from the computer-based assessment of problem-solving. These assessments may be especially useful when the cognitive outcomes of some innovative instructional methods are measured, such as project methods, problem-based learning and inquiry-based learning.

In the present study, we go further when we explore the possibilities for assessing domain-general problem-solving. Our test is based on the MicroDYN approach (Greiff and Funke, 2009 ; Funke and Greiff, 2017 ), which measures the same construct that was measured with several dynamic items in the PISA 2012 assessment (OECD, 2014 ) and in several other studies (Abele et al., 2012 ; Molnár et al., 2013 , 2017 ; Frischkorn et al., 2014 ).

DPS tasks have the same general characteristics. Simulated systems are presented which are based on practical contexts and situations that are easy for the problem-solver to comprehend. The simulated systems show a well-defined behavior, the problem-solver has to manipulate some input variables, and the system responds with changes in output variables. This represents a major difference over paper-based tests, as this sort of a realistic interaction with a responding system cannot be created on paper. The purpose of the interaction is to comprehend the rules that determine the behavior of the system.

In the first phase of completing a DPS test task, students interact with the simulated system, manipulate the values of independent variables, and observe how the changes impact the values of dependent variables. This interactive observation is the knowledge acquisition phase (also referred to as the rule identification phase), after which students depict the results of their observations on a concept map. Then, they have to manipulate the variables so that they reach a goal state; this is the knowledge utilization phase (or rule application phase). The results from the two phases are scored separately, and as previous research (e.g., Wüstenberg et al., 2012 ) has shown, there may be significant differences in performance in the two phases. These dynamic tasks are easily scalable, as the number of input and output variables as well as the relationships between them can be changed.

Aims and research questions

In the present study, we explore the prospects and value of assessing DPS in higher education. Such a test could later be a useful component of university admissions processes, especially in STEM disciplines, where studies require problem-solving in a technology-rich environment. The context of the study allows for an examination of the relationships between subject matter knowledge and problem-solving.

RQ1: How do matriculation examination results predict problem-solving test performance assessed at the beginning of higher education studies? We assume that the knowledge students possessed at the end of their high school studies (assessed by the matriculation exams) correlates with problem-solving, but the strengths of the relationships with the different domains is still open.
RQ2: What are the relationships between subject matter tests and problem-solving performance measured at the beginning of higher education studies? We assume that although the disciplinary knowledge tests correlate well with problem-solving, problem-solving measures other aspects of knowledge; therefore, it has considerable added value over subject matter tests.
RQ3: Are there differences between students in different disciplines? We assume that those who study within different divisions at the university also differ in their problem-solving skills and in the relationships between problem-solving and other tests.
RQ4:How do students' characteristics and background variables influence their problem-solving performance? We may assume that students' family background, mother's level of education, students' intention to learn and students' learning strategies influence how their problem-solving skills develop.

We expect that the results from these analyses may contribute to improving matriculation examinations as well as to devising better admissions processes.

Participants

Participants in the study were students admitted to a large Hungarian university and starting their studies. The university has 12 divisions (arts, science, medicine, etc.), but they vary in size (number of students). All of the divisions participated in the study, but because of the differences between them, not all analyses are equally relevant for every division.

The population for the study was formed exclusively of students who had just finished their high school studies and immediately applied for admission to the university. They took their matriculation examinations in May, and the assessment for this study was carried out in September of the same year.

The target population was 2,319 students, of whom 1,468 (63.3%) participated in the assessment; 57.7% of them were female. The participation rate by division varied from 28.18 to 74.16%.

Student participation was voluntary; students were notified of their option to take part in the assessment prior to commencing their studies. As an incentive, they received credits for successful completion of the tests.

Instruments

Problem-solving test.

Students completed a DPS test based on the MicroDYN approach. Several tests composed of similar tasks based on this model have already been used in other studies in Hungary (Molnár et al., 2013 , 2017 ) but only with younger participants. The test prepared for this study consisted of 20 items with varying difficulty levels.

For example, in the knowledge acquisition phase of an easy item, students had to observe how changing the values of two independent variables (e.g., two different kinds of syrup) impacted the value of one dependent (target) variable (sweetness of the lemonade). They moved sliders on the screen to set the current value for the blue and for the green syrup. The system responded by indicating the resultant sweetness level. Students observed what happened and attempted a new setting, observing the sweetness level with such a setting. They had 180 s for the knowledge acquisition phase in each task. In the knowledge utilization phase, they had to reach the required value of the dependent variable (sweetness) by setting the proper values of the independent variables in no more than 180 s. In a difficult item, students had to comprehend more complex relationships between three independent variables (three different training methods used by basketball players) and three dependent variables (motivation, power of the throw and exhaustion). (For more examples of similar DPS items, see Greiff et al., 2013b ; OECD, 2014 ). The two phases of problem-solving were scored separately. The score for the first (knowledge acquisition) phase was based on how accurately the relationships between the variables were depicted, while the score for the second (knowledge utilization) phase reflected the success with which the dependent variables reached the target state.

The difficulty level of the test was close to the optimal for the whole sample with a 45% mean (SD = 21.74). The reliability (Cronbach's alpha) of the entire test was 0.88. The reliabilities of the two problem-solving phases were also high (knowledge acquisition: 0.84; knowledge utilization: 0.83).

Disciplinary knowledge tests

Five disciplinary knowledge tests were prepared for the assessment: Hungarian language and literature (Hungarian, for short, with a strong reading comprehension component), mathematics, history, science, and English as a Foreign Language (EFL). Test content was based on the students' high school studies. The tests covered the major topics of the particular disciplines. Difficulty levels for the tests were adjusted approximately to the intermediate-level standards of the matriculation examination. These tests were prepared by experts practiced in preparing matriculation examination tests. The tests made use of the options made available by computer-based testing; using a variety of stimuli (e.g., texts, images, and animation) and response capture (e.g., entering texts and numbers, clicking, and moving objects on the screen by drag-and-drop). The descriptive statistics for the entire sample and the reliability of the instruments are summarized in Table Table1. 1 . The reliability coefficients for the tests were good, ranging from 0.88 to 0.96.

Disciplinary knowledge test: descriptive statistics and reliability coefficients.

Background questionnaire

A background questionnaire was administered to participating students via the same platform as the tests. Data were collected in this way about their matriculation examination results and their learning strategies and SES. To minimize the time devoted to administering the questionnaire, only the most relevant variables were explored, where strong relationships were expected. Family background was represented by mothers' level of education (from primary school to master's degree). Students' commitment to study (intention to learn) was measured with the highest degree they intend to earn (bachelor's, master's or PhD). Two scales for learning strategies that use self-reported Likert scales were adapted from the PISA 2000 assessment (elaboration strategies and memorization strategies, see Artelt et al., 2003 ).

The assessments were carried out in a large computer room at the university learning and information center. Three 2-h sessions (1 h per test) were offered to the students in the first 2 weeks of the semester. The tests and the questionnaire were administered using the eDia online platform.

Students received detailed feedback on their performance a week after the testing period ended. The feedback contained detailed analyses of their performance in the context of normative comparative data.

Data from the achievement tests were analyzed with IRT models. Plausible values were computed to compare the achievements of the age groups, and Weighted Likelihood Estimates (WLE) were used to compute person parameters. The analyses were performed with the ACER ConQuest program package (Wu et al., 1998 ). Person parameters were transformed to a 500(100) scale so that the university means were set to 500. MPlus software was used to conduct the structural equation modeling (Muthén and Muthén, 2012 ).

In this section, we first answer the research questions by examining the details of the correlations between subject matter knowledge represented in the matriculation examination results and in the test scores from the beginning of studies in higher education. Then, we synthesize the relationships in a path model based on these findings.

Matriculation examination results as predictors of problem-solving performance

Performance in two phases of problem-solving (knowledge acquisition and knowledge utilization) correlated at the moderate level ( r = 0.432, p < 0.001); therefore, it is worth examining the correlations between the matriculation examination results and the phases of problem-solving separately. Here, we only deal systematically with the three mandatory matriculation examination subjects, as these data are available for all participants, while only a small proportion of students took the exams in a science discipline or EFL as an elective. As few students took the matriculation examinations at the advanced level, this analysis involves the results from the intermediate exams. For a comparison, we have computed the correlations between matriculation examination results and those from the knowledge tests (see Table Table2 2 ).

Correlations between the matriculation examination results and those from the tests administered at the beginning of higher education studies.

Two major observations stand out from Table Table2. 2 . First, the mathematics matriculation result (which is based on a paper-and-pencil test with constructed responses) predicts problem-solving much more strongly than those in the two other subjects. Second, knowledge acquisition has a stronger correlation with the matriculation examination results than knowledge utilization does. The mathematics and history matriculation results predict the test results for the same respective subjects well; they are lower for Hungarian, which has no significant correlation with problem-solving. We note that when comparing the correlations, ca.0.05 differences are significant at p < 0.05, while ca.0.1 differences are significant at p < 0.001 (one-tailed, calculated by the Fisher r-to-z transformation). When we note differences between correlations, they are statistically significant.

Relationships between subject matter tests and problem-solving

Correlations for the six tests are summarized in Table Table3. 3 . The correlations between disciplinary knowledge test results are moderate (Hungarian and history with science and EFL) or large, and as expected from the similarities between these subjects, the Hungarian–history and mathematics–science pairs correlate more strongly than other pairs. Mathematics has the strongest correlation with problem-solving, followed by science.

Correlations for the tests taken at the beginning of higher education studies.

These correlations confirm once again that knowledge acquisition is a more decisive component of problem-solving items than knowledge utilization. To examine the details of this relationship, we performed regression analyses with problem-solving and its two phases as dependent variables, using the disciplinary knowledge test results as independent variables (Table (Table4 4 ).

Regression analyses of problem-solving and its two phases as dependent variables with disciplinary knowledge tests as independent variables.

The differences between these analyses confirm previous observations on the role of knowledge acquisition and indicate that it is only mathematics and science whose contribution to the variance explained is significant and positive. Furthermore, even in the cases of knowledge acquisition, ~70% of the variance remained unexplained.

Differences between students studying within different divisions

As can be expected, there are large differences between the divisions at the university, both in performance on knowledge tests and on problem-solving. Therefore, it is anticipated that problem-solving has different relationships with disciplinary knowledge. To examine these differences, we have chosen two divisions with a large number of students participating in the assessments and with different study profiles. The division that deals with the humanities, known as the Faculty of Arts (Arts, for short), participated with 212 students (65.2% of the population, 71.7% female), and the division that deals mainly with the natural sciences, known as the Faculty of Science and Informatics (Science, for short), was represented with 380 students (64.0% of the population, 32.8% female). They performed differently on each test (Table (Table5), 5 ), including problem-solving.

Differences in achievement among students in the two divisions with different study profiles.

Achievement differed according to the expectations for the different study profiles. Students at the Arts Faculty performed better in Hungarian, history and EFL, while Science Faculty students performed better in mathematics, science and problem-solving.

To examine the details of the relations between disciplinary knowledge and problem-solving, we performed the regression analyses separately for the two divisions. Taking into account the decisive role of knowledge acquisition, we present only the results for this phase of problem-solving in Table Table6. 6 . For comparison, the R 2 were 0.203 (Arts) and 0.217 (Science) for the entire problem-solving test when the same analyses were performed.

Regression analyses of knowledge acquisition as a dependent variable with disciplinary knowledge tests as independent variables for the two divisions.

Although the same amount of variance of knowledge acquisition was explained by the same set of independent variables, the contributions of the individual variables are different. Mathematics and science play an important role at both divisions, and the contribution of EFL is also significant at the Faculty of Science.

Relationships between students' background variables and problem-solving performance

Previous studies (e.g., OECD, 2014 ) have indicated large difference in problem-solving in a number of dimensions. Here, we explore the differences according to some available background variables.

Gender differences

Gender differences are routinely analyzed on large-scale national and international assessments. The PISA studies indicated that Hungarian girls' reading comprehension was significantly better than that of boys, while boys' performance was better in mathematics and there were no significant gender differences in science (OECD, 2016 ). Female and male students performed differently on problem-solving in this study as well. To provide context to interpret the size of gender difference in problem-solving, the differences on other tests are also indicated in Table Table7 7 .

Gender differences in test performance.

All differences are significant at p < 0.01 .

The only test where women outperform men was Hungarian language (in line with the better reading performance of the female students); on all other tests, men performed better. The largest difference was found in favor of men in problem-solving. Here again, knowledge acquisition shows a much larger difference, indicating that this is the more sensitive phase of problem-solving.

Mothers' education and intention to learn

The relationship of test performance with students' socio-economic status is a well-known phenomenon, although there are large differences in this respect between countries and also between domains of assessment. International assessment programs (e.g., the PISA studies) usually involve complex indices for this purpose, but we have only one variable to represent students' family background, mothers' education. A further variable that may be interesting in this context is what degree students want to earn (intention to learn). We have found a small (Spearman's rho = 0.182, p < 0.001) correlation between these two variables. The correlations of test results with mother's educational level and intention to learn are summarized in Table Table8 8 .

Correlations of performance on the tests with mother's education and intention to learn.

There are no large differences between the correlations; all are rather small. Mothers' education has little impact on problem-solving. The correlation of problem-solving with intention to learn is small but still significant; the correlation with knowledge utilization here is also smaller than with knowledge acquisition.

Learning strategies

As there are only a few questions in the learning strategies questionnaire, we present the texts and the correlations with the problem-solving achievement for each question. Students' answers to these questions show small but significant correlations with problem-solving (Table (Table9 9 ).

Correlations of the learning strategies questions with problem-solving performance.

DPS1, Knowledge acquisition; DPS2, Knowledge utilization; DPS, dynamic problem-solving .

The elaboration strategies questions correlate positively with problem-solving, while the memorization strategy questions correlate negatively with it. It is quite clear from the content of the questions that students who prefer conceptual meaningful learning over rote learning are better problem-solvers.

An integrated model of the relations of knowledge acquisition in dynamic problem-solving

We synthesized the results using structural equation modeling (SEM). Taking into account the observations reported in the previous sections, here we deal only with the knowledge acquisition phase. As the main aim of the present study is to validate the DPS test and to explore its usefulness at the beginning of university studies, we conceived a model by using variables with significant correlations. We assume that students' gender and learning strategies influence their disciplinary test results, while these results (students' actual knowledge) influence achievement in DPS.

A model that adequately fits the data (RMSEA = 0.046, CFI = 0.986, TLI = 0.949) is presented in Figure Figure1. 1 . Gender influences mathematics test results, while learning strategies have a remarkable impact on mathematics and science. These two disciplines and history have a significant relationship with the first phase of problem-solving.

An external file that holds a picture, illustration, etc.
Object name is fpsyg-08-02022-g0001.jpg

A path model of the relationships with the first phase of problem-solving (knowledge acquisition).

In this model, positive impacts of elaboration strategies on mathematics and science were found, while success in the knowledge acquisition phase of DPS was positively influenced by science and mathematics. Gender and memorization strategy as well as history have a negative relationship.

Our results confirmed or extended several findings from previous research (e.g., different relationships of the phases of problem-solving) and identified some new relationships as well (e.g., the relationships with learning strategies).

Determinants of problem-solving achievement at the beginning of higher education studies

Previous research has already identified several characteristics of DPS at different ages, including primary and secondary school students (Molnár et al., 2013 ), 15-year-old students in the PISA 2012 assessment (which used tests partially built on the MicroDYN approach in a large-scale international comparative survey, OECD, 2014 ), and university students (Wüstenberg et al., 2012 ). The present study has shown the feasibility and usefulness of such an assessment in higher education, indicating that DPS is an easily applicable test with several characteristics of the twenty-first-century skills.

Based on the available data, the impact of previous learning was represented by the disciplinary knowledge test results of three matriculation examination subjects. Mathematics had the strongest correlation with problem-solving, which can be explained by the fact that mathematics is studied throughout the 12 years of primary and secondary schooling and by the nature of cognitive processes required by problem-solving (Greiff et al., 2013b ; Csapó and Funke, 2017b ). The important role of mathematics was also noticed when the correlations with the subject tests were analyzed, and the integrating path model mirrored the same exceptional impact as well.

The first phase of solving dynamic items (knowledge acquisition or rule identification in other studies) has a stronger relationship with any other observed variable than the second phase (knowledge utilization or rule application). Other studies have found similar differences, although the dominance of knowledge acquisition was not so obvious (Wüstenberg et al., 2012 ). The important role of the first phase, indicated by larger correlations, may be attributed to the kind of reasoning this phase requires. Students have to combine the different values of the independent variables they manipulate in this phase (combinatorial reasoning), judge certain probabilities (probabilistic reasoning) and abstract rules from the observed behavior of the simulated system (inductive reasoning). This may also explain the strong connection (especially of the first phase) to mathematics, as this kind of reasoning is mostly applied when learning mathematics. Rule induction connects DPS to general intelligence as well, as most intelligence tests use inductive reasoning items. Nevertheless, previous research has indicated that problem-solving explains added variance of students' school achievement (GPA) beyond intelligence tests (Wüstenberg et al., 2012 ), and moderate to large correlation ( r = 0.44, 0.52, and 0.47 in Grades 5, 7, and 11) has been found between problem-solving and inductive reasoning (Molnár et al., 2013 ).

Our analyses showed that there were differences between the students preparing for studies in different disciplines both on the level of problem-solving achievement and in the strengths of correlations with domain-specific knowledge. However, some main tendencies, e.g., the dominant role of mathematics and science and the role of the knowledge acquisition phase, may be generalized.

Large gender differences were found on all the tests we used in this study, but the largest one was observed in problem-solving (78 points), mathematics being the second largest one with a much lower difference (49 points). The difference in knowledge acquisition is especially high (93 points). In PISA 2012, gender differences in problem-solving varied from country to country. The OECD mean was 7 points, and in Hungary it was below average, though it was not significant, only 3 points (OECD, 2014 ). To interpret this discrepancy between the PISA results and the present study, it is worth noting that the Hungarian PISA problem-solving results were below average (459 points) and that not all items were dynamic. Furthermore, in our study, women are overrepresented in the Arts Faculty and in humanities studies in general, while they are underrepresented in STEM studies.

Although there are large differences between students according to the socio-economic status (SES) of their family—and Hungary belongs to a group of countries where the impact of SES is especially strong—there was no large effect found in problem-solving in the PISA 2012 survey. In our study, we also found a modest impact of mothers' level of education on students' problem-solving performance. The fact that problem-solving is less determined by social background than domain-specific competencies indicates a potential opportunity for disadvantaged students as they may show their strengths on these kinds of assessments.

Previous studies have indicated a strong relationship between low- and high-achieving high school students and the different learning strategies they use (Yip, 2013 ). Our results confirm this notion, as there are clear links between learning strategies and knowledge acquisition in problem-solving. A positive effect of elaboration strategies may have been predictable, but a measurable negative impact of memorization strategies is somewhat unexpected. These results suggest the conclusion that problem-solving is learnable and point to one of the directions in the search for proper training methods. In general, there are two main directions for facilitating the development of this kind of general cognitive skill in a school context. The first is a holistic approach, when developmental impacts are embedded in other educational activities, in this case in learning science and mathematics through meaningful elaborative strategies. Discovery learning and inquiry-based teaching methods may have an impact on the development of problem-solving as well. The second method improves problem-solving by developing component skills (Csapó and Funke, 2017b ). We have identified potential component skills; providing training in them may also influence the development of problem-solving.

The results of the SEM indicate the complex nature of the relationships between the variables being explored. The DPS tasks are constructed so that completing them requires no preliminary knowledge within any discipline. Therefore, we may assume that if there are relationships between disciplinary knowledge tests and DPS tests, these relationships are established by factors other than the factual knowledge represented in the knowledge tests. Such factors may be learning strategies (we have variables for representing them) and certain cognitive skills needed both for completing the disciplinary tests and the DPS tests (in this study, we have no variables to represent them in the SEM). In this model, gender as a variable (most probably) mediates women's better reading and poorer mathematics achievement (shown by other studies, e.g., PISA). In sum, this model indicates that men outperform women, and this impact is mediated by the higher mathematics performance among men. The negative impact of memorization is transmitted via mathematics and science.

Limitations of the present study

As the PISA 2012 assessments also indicated, there are large differences between countries not only on the level of problem-solving performance, but also in the strengths of the relationships between several relevant variables as well; therefore, some particular results found in one country cannot be generalized over countries and cultures. Although some general tendencies were found, we have seen that the strength of the relationships we have examined in this study differs by division. Therefore, the generalizability of the strengths of these relationships is limited; nevertheless, the method we applied in this study is generalizable and may be useful to explore the actual relationships in any higher educational context. Participation was voluntary in the study; the actual samples are thus not representative of the divisions. Nevertheless, the analyses revealed some generalizable tendencies as well.

Conclusions: further research and prospects for assessment supporting high school–university transition

The results from the present study have raised several further questions worth researching. The dominant role of knowledge acquisition indicates a promising line of inquiry to explore this phase in more detail. One promising direction is to identify students' knowledge acquisition strategies, e.g., the way they manipulate the independent variables when they attempt to discover how these manipulations impact changes in the dependent variable. Students' activities are logged, and their strategies may be ascertained with log file analyses. Latent class analysis may be an effective method to identify students' exploring strategies.

The knowledge acquisition phase also deserves further study from the perspective of its relationships to learning strategies as well, for example, examining if poor problem-solving performance can be an indicator of inadequate learning strategies. If such a connection can be proven, problem-solving assessment could be a diagnostic tool for identifying poor leaning strategies, possibly more reliable than self-reported questionnaires. Further insights into the nature of cognition in the knowledge acquisition phase could be expected from studying it in relation to the learning to learn assessments (Greiff et al., 2013b ; Vainikainen et al., 2015 ).

Several skills may be identified which are needed to successfully complete phases of problem-solving. A systematic examination of the role of some supposed component skills (e.g., combinatorial reasoning, probabilistic reasoning, correlational reasoning and inductive reasoning) would provide foundations for the development of problem-solving by strengthening its component skills as well.

The results from this study indicated that technology-based assessment of problem-solving may be a useful instrument to moderate the secondary–tertiary education transition. To improve its usefulness, the scoring system may be further developed, extending it with an automated log file analysis. Such an instrument would be especially helpful in selection processes (admissions tests) for the STEM disciplines. More detailed analyses of the relationship between problem-solving and the study profile would be needed to improve the test. In the present study, we compared divisions of study within the university, but a division is still not homogeneous; for example, students in biology training may be different from those in mathematics.

We have found significant positive relationships with the questions on elaboration learning strategies and negative relationships with the questions on memorization strategies. In the present study, there were not enough questions to use sophisticated scales for representing these learning strategies, but the findings indicate the relevance of exploring the role learning strategies play in the development of problem-solving. This seems a promising area both for research and practice not only for higher education but also for earlier phases of school education.

The predictive power of DPS can be explored later when data is available on the university achievement of the students participating in the present assessment. The test may have a diagnostic value (indicating poor study strategies or insufficient problem-solving skills) and can also be used to aid students in selecting a study track better suited to their cognitive skills.

Author contributions

BC and GM have made an equal contribution to the study, including the design, data collection, analyses, and writing of the manuscript, and have both approved it for publication.

Conflict of interest statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest. The reviewer, CL, and handling Editor declared their shared affiliation.

Funding. This study was completed within the research program of the MTA–SZTE Research Group on the Development of Competencies. The data analysis and preparation of the manuscript were funded by OTKA K115497.

Abele S., Greiff S., Gschwendtner T., Wüstenberg S., Nickolaus R., Nitzschke A., et al. (2012). Dynamische Problemlösekompetenz. Ein bedeutsamer Prädiktor von Problemlöseleistungen in technischen Anforderungskontexten? [Dynamic problem solving – an important predictor of problem-solving performance in technical domains?] . Zeitschrift für Erziehungswissenschaft 15 , 363–391. 10.1007/s11618-012-0277-9 [ CrossRef ] [ Google Scholar ]
Adey P., Csapó B., Demetriou A., Hautamaki J., Shayer M. (2007). Can we be intelligent about intelligence? Why education needs the concept of plastic general ability . Educ. Res. Rev. 2 , 75–97. 10.1016/j.edurev.2007.05.001 [ CrossRef ] [ Google Scholar ]
Artelt C., Baumert J., Julius-McElvani N., Peschar J. (2003). Learners for Life. Student Approaches to Learning. Results form PISA 2000 . Paris: OECD. [ Google Scholar ]
Atkinson R. C., Geiser S. (2009). Reflections on a century of college admissions tests . Educ. Res. 38 , 665–676. 10.3102/0013189X09351981 [ CrossRef ] [ Google Scholar ]
Binkley M., Erstad O., Herman J., Raizen S., Ripley M., Miller-Ricci M., et al. (2012). Defining twenty-first century skills , in Assessment and Teaching of 21st Century Skills , eds Griffin P., McGaw B., Care E. (New York, NY: Springer; ), 17–66. [ Google Scholar ]
Blech C., Funke J. (2005). Dynamis review: An overview about Applications of the Dynamis Approach in Cognitive Psychology. Bonn: German Institute for Adult Education (DIE) . Available online at: http://www.die-bonn.de/esprid/dokumente/doc-2005/blech05_01.pdf
Bryan C., Clegg K. (eds.). (2006). Innovative Assessment in Higher Education. Abingdon: Routledge. [ Google Scholar ]
Buchner A., Funke J. (1993). Finite-state automata: dynamic task environments in problem-solving research . Q. J. Exp. Psychol. 46 , 83–118. 10.1080/14640749308401068 [ CrossRef ] [ Google Scholar ]
Coates H. (2016). Assessing student learning outcomes internationally: insights and frontiers . Asses. Eval. High. Educ. 41 , 662–676. 10.1080/02602938.2016.1160273 [ CrossRef ] [ Google Scholar ]
Csapó B. (2009). The second stage of public education and the Matura , in Green Book for the Renewal of Public Education in Hungary , eds Fazekas K., Köllo J., Varga J. (Budapest: Ecostat; ), 81–104. [ Google Scholar ]
Csapó B., Ainley J., Bennett R., Latour T., Law N. (2012). Technological issues of computer-based assessment of 21st century skills , in Assessment and Teaching of 21st Century Skills , eds Griffin P., McGaw B., Care E. (New York, NY: Springer; ), 143–230. [ Google Scholar ]
Csapó B., Funke J. (eds.) (2017a). The Nature of Problem Solving. Using Research to Inspire 21st Century Learning. Paris: OECD. [ Google Scholar ]
Csapó B., Funke J. (2017b). The development and assessment of problem solving in 21st-century schools , in The Nature of Problem Solving. Using Research to Inspire 21st Century Learning , eds Csapó B., Funke J. (OECD: Paris; ), 19–32. [ Google Scholar ]
Fischer A., Greiff S., Funke J. (2017). The history of complex problem solving , in The Nature of Problem Solving. Using Research to Inspire 21st Century Learning , eds Csapó B., Funke J. (OECD: Paris; ), 107–121. [ Google Scholar ]
Fleischer J., Buchwald F., Leutner D., Wirth J., Rumann S. (2017). Analytical problem solving: Potentials and manifestations , in The Nature of Problem Solving. Using Research to Inspire 21st Century Learning , eds Csapó B., Funke J. (OECD: Paris; ), 33–45. [ Google Scholar ]
Frischkorn G. T., Greiff S., Wüstenberg S. (2014). The development of complex problem solving in adolescence: a latent growth curve analysis . J. Educ. Psychol. 106 , 1007–1020. 10.1037/a0037114 [ CrossRef ] [ Google Scholar ]
Funke J. (1988). Using simulation to study complex problem solving: a review of studies in the FRG . Simul. Games 19 , 277–303. 10.1177/0037550088193003 [ CrossRef ] [ Google Scholar ]
Funke J. (1993). Microworlds based on linear equation systems: a new approach to complex problem solving and experimental results . Adv. Psychol . 101 , 313–330. 10.1016/S0166-4115(08)62663-1 [ CrossRef ] [ Google Scholar ]
Funke J. (1998). Computer-based testing and training with scenarios from complex problem-solving research: advantages and disadvantages . Int. J. Sel. Asses . 6 , 90–96. 10.1111/1468-2389.00077 [ CrossRef ] [ Google Scholar ]
Funke J. (2001). Dynamic systems as tools for analysing human judgement . Think. Reason . 7 , 69–89. 10.1080/13546780042000046 [ CrossRef ] [ Google Scholar ]
Funke J. (2010). Complex problem solving: a case for complex cognition? Cogn. Process. 11 , 133–142. 10.1007/s10339-009-0345-0 [ PubMed ] [ CrossRef ] [ Google Scholar ]
Funke J. (2014). Analysis of minimal complex systems and complex problem solving require different forms of causal cognition . Front. Psychol . 5 :739. 10.3389/fpsyg.2014.00739 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
Funke J., Greiff S. (2017). Dynamic problem solving: Multiple-item testing based on minimally complex systems , in Competence Assessment in Education , eds Leutner D., Fleischer J., Grünkorn J., Klieme E. (Cham: Springer International Publishing; ), 427–443. [ Google Scholar ]
Greiff S., Funke J. (2009). Measuring complex problem solving: The MicroDYN approach , in The Transition to Computer-Based Assessment. New Approaches to Skills Assessment and Implications for Large-Scale Testing , eds Scheuermann F., Björnsson J. (Luxembourg: Office for Official Publications of the European Communities; ), 157–163. [ Google Scholar ]
Greiff S., Funke J. (2017). Interactive problem solving: Exploring the potential of minimal complex systems , in The Nature of Problem Solving. Using Research to Inspire 21st Century Learning , eds Csapó B., Funke J. (Paris: OECD; ), 93–105. [ Google Scholar ]
Greiff S., Wüstenberg S., Csapó B., Demetriou A., Hautamäki H., Graesser A. C., et al. (2014). Domain-general problem solving skills and education in the 21st century . Educ. Res. Rev . 13 , 74–83. 10.1016/j.edurev.2014.10.002 [ CrossRef ] [ Google Scholar ]
Greiff S., Wüstenberg S., Funke J. (2012). Dynamic problem solving: a new assessment perspective . Appl. Psychol. Meas. 36 , 189–213. 10.1177/0146621612439620 [ CrossRef ] [ Google Scholar ]
Greiff S., Wüstenberg S., Holt D. V., Goldhammer F., Funke J. (2013a). Computer-based assessment of complex problem solving: concept, implementation, and application . Educ. Technol. Res. Dev . 61 , 407–421. 10.1007/s11423-013-9301-x [ CrossRef ] [ Google Scholar ]
Greiff S., Wüstenberg S., Molnár G., Fischer A., Funke J., Csapó B. (2013b). Complex problem solving in educational contexts – something beyond g: concept, assessment, measurement invariance, and construct validity . J. Educ. Psychol. 105 , 364–379. [ Google Scholar ]
Griffin P., Care E. (eds.) (2015). Assessment and Teaching of 21st Century Skills. Methods and Approach. Dordrecht: Springer Netherlands. [ Google Scholar ]
Griffin P., McGaw B., Care E. (eds.) (2012). Assessment and Teaching of 21st Century Skills. New York, NY: Springer. [ Google Scholar ]
Hutchings P., Kinzie J., Kuh G. D. (2015). Evidence of student learning. What counts and what matters for improvement , in Using Evidence of Student Learning to Improve Higher Education , eds Kuh D., Ikenberry S. O., Jankowski N. A., Cain T. R., Ewell P. T., Hutchings P. H., Kinzie J. (San Francisco, CA: Jossey-Bass; ), 1–23. [ Google Scholar ]
Ikenberry S. O., Kuh G. D. (2015). From compliance to ownership: why and how colleges and universities assess student learning , in Using Evidence of Student Learning to Improve Higher Education , eds Kuh D., Ikenberry S. O., Jankowski N. A., Cain T. R., Ewell P. T., Hutchings P. H., Kinzie J. (San Francisco, CA: Jossey-Bass; ), 27–50. [ Google Scholar ]
Kong S. C., Chan T. W., Griffin P., Hoppe U., Huang R., Kinshuk L., et al. (2014). E-learning in school education in the coming 10 years for developing 21st century skills: critical research issues and policy implications . Educ. Technol. Soc . 17 , 70–78. [ Google Scholar ]
Lopez E. J., Shavelson R. J., Nandagopal K., Szu E., Penn J. (2014). Factors contributing to problem-solving performance in first-semester organic chemistry . J. Chem. Educ. 91 , 976–981. 10.1021/ed400696c [ CrossRef ] [ Google Scholar ]
Molnár G., Greiff S., Csapó B. (2013). Inductive reasoning, domain specific and complex problem solving: relations and development . Think. Skills Creat . 9 , 35–45. 10.1016/j.tsc.2013.03.002 [ CrossRef ] [ Google Scholar ]
Molnár G., Greiff S., Wüstenberg S., Fischer A. (2017). Empirical study of computer-based assessment of domain-general complex problem-solving skills , in The Nature of Problem Solving. Using Research to Inspire 21st Century Learning , eds Csapó B., Funke J. (OECD: Paris; ), 125–140. [ Google Scholar ]
Muthén L. K., Muthén B. O. (2012). Mplus User's Guide, 7th Edn . Los Angeles, CA: Muthén & Muthén. [ Google Scholar ]
National Research Council (2013). Education for Life and Work: Developing Transferable Knowledge and Skills in the 21st Century . Washington, DC: National Academies Press. [ Google Scholar ]
Neubert J. C., Mainert J., Kretzschmar A., Greiff S. (2015). The assessment of 21st century skills in industrial and organizational psychology: complex and collaborative problem solving . Ind. Organ. Psychol. 8 , 238–268. 10.1017/iop.2015.14 [ CrossRef ] [ Google Scholar ]
OECD (2004). Learning for Tomorrow's World: First Results from PISA 2003. Paris: OECD. [ Google Scholar ]
OECD (2013a). PISA 2012 Assessment and Analytic Framework . Paris: OECD [ Google Scholar ]
OECD (2013b). PISA 2015 Draft Collaborative Problem Solving Framework . Paris: OECD. [ Google Scholar ]
OECD (2014). PISA 2012 Results: Creative Problem Solving. Students' Skills in Tackling Real-Life Problems (Vol. V) . Paris: OECD. [ Google Scholar ]
OECD (2016). PISA 2015 Results: Excellence and Equity in Education (Vol. I). Paris: OECD. [ Google Scholar ]
Qian M., Clark K. R. (2016). Game-based learning and 21st century skills: a review of recent research . Comput. Hum. Behav. 63 , 50–58. 10.1016/j.chb.2016.05.023 [ CrossRef ] [ Google Scholar ]
Sonnleitner P., Brunner M., Greiff S., Funke J., Keller U., Martin R., et al. (2012). The genetics lab. Acceptance and psychometric characteristics of a computer-based microworld to assess complex problem solving . Psychol. Test Assess. Model. 54 , 54–72. [ Google Scholar ]
Tóth K., Rölke H., Goldhammer F., Barkow I. (2017). Educational process mining: New possibilities for understanding students' problem-solving skills , in The Nature of Problem Solving. Using Research to Inspire 21st Century Learning , eds Csapó B., Funke J. (Paris: OECD; ), 193–210. [ Google Scholar ]
Tremblay K., Lalancette D., Roseveare D. (2012). Assessment of Higher Education Learning Outcomes (AHELO) . Feasibility Study Report Vol. 1. Design and Implementation. Paris: OECD. [ Google Scholar ]
Trilling B., Fadel C. (2009). 21st Century Skills: Learning for Life in Our Times . San Francisco, CA: John Wiley & Sons. [ Google Scholar ]
Vainikainen M. P., Wüstenberg S., Kupiainen S., Hotulainen R., Hautamäki J. (2015). Development of learning to learn skills in primary school . Int. J. Lifelong Educ. 34 , 376–392. 10.1080/02601370.2015.1060025 [ CrossRef ] [ Google Scholar ]
Wiley E. W., Shavelson R. J., Kurpius A. A. (2014). On the factorial structure of the SAT and implications for next-generation college readiness assessments . Educ. Psychol. Meas. 74 859–874. 10.1177/0013164414528332 [ CrossRef ] [ Google Scholar ]
Wirth J., Klieme E. (2003). Computer-based assessment of problem solving competence . Asses. Educ. 10 , 329–345. 10.1080/0969594032000148172 [ CrossRef ] [ Google Scholar ]
Wu M., Adams M. J., Wilson M. (1998). ACER ConQuest [Computer Program and Manual] . Hawthorn, Australia: ACER Press. [ Google Scholar ]
Wüstenberg S., Greiff S., Funke J. (2012). Complex problem solving – more than reasoning? Intelligence 40 , 1–14. 10.1016/j.intell.2011.11.003 [ CrossRef ] [ Google Scholar ]
Yip M. C. W. (2013). Learning strategies and their relationships to academic performance of high school students in Hong Kong . Educ. Psychol. 33 , 817–827. 10.1080/01443410.2013.794493 [ CrossRef ] [ Google Scholar ]
Zlatkin-Troitschanskaia O., Shavelson R. J., Kuhn C. (2015). The international state of research on measurement of competency in higher education . Stud. High. Educ . 40 , 393–411. 10.1080/03075079.2015.1004241 [ CrossRef ] [ Google Scholar ]

IMAGES

Developing Problem-Solving Skills for Kids
(PDF) Problem-solving Ability Test (PSAT)
10 Problem Solving Skills Examples: How To Improve
Introduction to Problem Solving Skills
Problem Solving Test Practice: Questions and Answers!
test of problem solving subtests

VIDEO

Problem Solving|| Employability Skills|| ITI|| Module-03 BECOMING A PROFESSIONAL IN THE 21ST CENTURY
Introduction to problem solving
Problem Solving IQ Pedestal
NMMS Exam
समस्या समाधान योग्यता परीक्षण( problem solving ability test) PSAT-D test In PSYCHOLOGY
Find the Deferens & improve your brain skills 【#004, Difficulty Medium】| Quiz Brainly

COMMENTS

What is Problem Solving? An Introduction
Problem solving, in the simplest terms, is the process of identifying a problem, analyzing it, and finding the most effective solution to overcome it. For software engineers, this process is deeply embedded in their daily workflow. It could be something as simple as figuring out why a piece of code isn't working as expected, or something as ...
How to assess problem-solving skills
Communication, listening, and customer service skills. In all the stages of problem-solving, you need to listen and engage to understand what the problem is and come to a conclusion as to what the solution may be. Another challenge is being able to communicate effectively so that people understand what you're saying.
Introduction to Problem Solving Skills
Good problem solving skills empower you not only in your personal life but are critical in your professional life. In the current fast-changing global economy, employers often identify everyday problem solving as crucial to the success of their organizations. For employees, problem solving can be used to develop practical and creative solutions ...
Problem Solving Skills Test
Step 1: Find the Problem. (Questions 7, 12) Your score is 0 out of 0. Some problems are very obvious, however others are not so easily identified. As part of an effective problem-solving process, you need to look actively for problems - even when things seem to be running fine.
What Are Problem-Solving Skills? Definition and Examples
Problem-Solving Skills Definition. Problem-solving skills are the ability to identify problems, brainstorm and analyze answers, and implement the best solutions. An employee with good problem-solving skills is both a self-starter and a collaborative teammate; they are proactive in understanding the root of a problem and work with others to ...
Test Your Problem-Solving Skills
Test Your Problem-Solving Skills. Personalize Your Emails Personalize your monthly updates from BrainFacts.org by choosing the topics that you care about most! Sign Up Find a Neuroscientist Engage local scientists to educate your community about the brain. ...
How Good Is Your Problem Solving?
Enjoy exploring these stages! Step 1: Find the Problem (Questions 7, 12) Some problems are very obvious, however others are not so easily identified. As part of an effective problem-solving process, you need to look actively for problems - even when things seem to be running fine.
Art of Problem Solving
Identify the Problem: The very first step in problem solving is to identify the problem. Problem solving is harder if one doesn't know what "problem" to solve! This is often seen in engineering and math by reading the problem statement. Make a Game Plan: The next step is to have a plan on what to do. In other words, just rushing in won ...
What is Problem Solving? Steps, Process & Techniques
Finding a suitable solution for issues can be accomplished by following the basic four-step problem-solving process and methodology outlined below. Step. Characteristics. 1. Define the problem. Differentiate fact from opinion. Specify underlying causes. Consult each faction involved for information. State the problem specifically.
Problem solving (video)
The video explains different problem-solving methods, including trial and error, algorithm strategy, and heuristics. It also discusses concepts like means-end analysis, working backwards, fixation, and insight. These techniques help us tackle both well-defined and ill-defined problems effectively. Created by Carole Yue.
PDF Problem Solving Ability Assessment Based on Design for Secondary ...
Introduction Problem solving ability has played a critical role in human history (Chi & Glaser, 1985; ... Chang (2010) developed a problem solving ability test (PSAT) using open-ended essay-questions based on the creative problem solving model of Osborn (as cited in Chang, 2010, p. 106). The PSAT determines the level of students'
Creative Problem-Solving Test
This Creative Problem-solving Test was developed to evaluate whether your attitude towards problem-solving and the manner in which you approach a problem are conducive to creative thinking. This ...
7.3 Problem Solving
Solving Puzzles. Problem-solving abilities can improve with practice. Many people challenge themselves every day with puzzles and other mental exercises to sharpen their problem-solving skills. Sudoku puzzles appear daily in most newspapers. Typically, a sudoku puzzle is a 9×9 grid. The simple sudoku below is a 4×4 grid. To solve the puzzle ...
McKinsey Problem Solving Test Practice Test A
This practice test has been developed to provide a sample of the actual McKinsey Problem Solving Test used for selection purposes. This test assesses your ability to solve business problems using deductive, inductive, and quantitative reasoning. This practice test contains a total of 26 questions. The actual test contains 26
Ability Test
Ability tests, also known as aptitude tests, assess various cognitive abilities such as verbal, numerical, abstract reasoning, spatial awareness, logical reasoning, and problem-solving. These tests are designed to evaluate individuals' natural talents and potential to learn and succeed in specific areas. For example, a verbal ability test ...
Construction and Validation of Problem-Solving Ability Test
A 16 x 14 puzzle box was used to test problem solving ability. F-statistics showed significant main effects of cognitive style (p < .001) and gender (p < 001) on problem solving.
PDF INTRODUCTION TO PROBLEM SOLVING
Problem-Solving strategies 1-6 Strategies 1. guess and Test 2. Draw a Picture 3. Use a variable 4. look for a Pattern 5. make a list 6. Solve a Simpler Problem Because problem solving is the main goal of mathematics, this chapter introduces the six strategies listed in the Problem-Solving Strategies box that are helpful in solving problems.
Module 7: Thinking, Reasoning, and Problem-Solving
7. Module 7: Thinking, Reasoning, and Problem-Solving. This module is about how a solid working knowledge of psychological principles can help you to think more effectively, so you can succeed in school and life. You might be inclined to believe that—because you have been thinking for as long as you can remember, because you are able to ...
Problem Solving Ability Test
Pr oblem Solving Ability T est (PSA T) Problem Solving Ability T e st (PSA T), developed by L.N. Dubey and C.P. Mathur, enables us. to measure the problem solving ability of the participant. Problem solving a bility is highly. correlated with intelligence, reasoning ability and mathematical abil ity. This test has a.
Thinking and Intelligence
6. Thinking and Intelligence. Figure 7.1 Thinking is an important part of our human experience, and one that has captivated people for centuries. Today, it is one area of psychological study. The 19th-century Girl with a Book by José Ferraz de Almeida Júnior, the 20th-century sculpture The Thinker by August Rodin, and Shi Ke's 10th-century ...
Learning Mathematics Problem Solving through Test Practice: a
We measure the effect of a single test practice on 15-year-old students' ability to solve mathematics problems using large, representative samples of the schooled population in 32 countries. We exploit three unique features of the 2012 administration of the Programme for International Student Assessment (PISA), a large-scale, low-stakes international assessment. During the 2012 PISA ...
Self-Assessment in the Development of Mathematical Problem-Solving Skills
In fact, problem-solving deeply requires meta-cognitive skills [8,9], and reflection on one's work has been pointed out as one of the promotional approaches to teach problem solving [10,11]. Metacognition and reflection are at the core of self-assessment processes [ 2 ].
Potential for Assessing Dynamic Problem-Solving at the Beginning of
Introduction. The social and economic developments of the past decades have re-launched the debate on the mission of schooling, more specifically, on the types of skills schools are expected to develop in their students in order to prepare them for an unknown future. ... (Arts) and 0.217 (Science) for the entire problem-solving test when the ...
Students Hone Problem-Solving Skills in Crisis Simulation Exercise
Over the weekend of February 23-25, 2024, Johns Hopkins SAIS held its annual school-wide Crisis Simulation exercise, bringing together more than 70 students in an experiential learning activity to test their critical thinking and problem-solving skills as they tried to resolve a theoretical global crisis. The exercise was the culmination of six ...

What Is Problem Solving? How Software Engineers Approach Complex Challenges

What Is Problem Solving?

The Importance of Problem-Solving Skills for Software Engineers

Driving Development Forward

Innovation and Optimization

Increasing Efficiency and Productivity

Improving Software Quality

Problem-Solving Techniques in Software Engineering

Decomposition

Abstraction

Algorithmic Thinking

Parallel Thinking

Testing and Validation

Evaluating Problem-Solving Skills

Recognizing Problem-Solving Skills in Candidates

Assessing Problem-Solving Skills

Key Takeaways

Get started with HackerRank

Recommended topics

What Factors Actually Impact a Developer’s Decision to Accept an Offer?

How to assess problem-solving skills

Introduction to key problem-solving skills

Communication, listening, and customer service skills

Data analysis, research, and topic understanding skills

Dependability, believability, trustworthiness, and follow-through

Leadership, team-building, and decision-making

Are you looking for a holistic way to develop leaders in your workplace?

Why is problem solving important in the workplace?

Proven methods to assess and improve problem-solving skills

Convergent and divergent thinking methods

Starbursting

Use of data analysis to measure improvement in problem-solving skills for your organization

Stay up to date with our newsletter

More stories we think you will like

How to measure your succession planning strategy: 4 metrics you need to track

External vs internal hiring: pros and cons of every method

The importance of design thinking and Agile methods in today’s HR

Not a customer yet? Contact us

Career at HRForecast

Introduction to Problem Solving Skills

What Does Problem Solving Look Like?

Watch the video

Developing Problem Solving Processes

Voices From the Field: Solving Problems

Meet the Partners:

Making Decisions

Root Cause Analysis

How Huddles Work

Brainstorming

Importance of Good Communication and Problem Description

Methods of Communication

Summary of Strategies

Problem Solving: An Important Job Skill

Quick Summary

How Good Is Your Problem Solving?

How Good Are You at Solving Problems?

Your last quiz results are shown.

Score Interpretation

Figure 1 – The Simplexity Thinking Process

Step 1: Find the Problem

Step 2: Find the Facts

Step 3: Define the Problem

Step 4: Find Ideas

Step 5: Select and Evaluate

Step 6: Plan

Step 7: Sell the Idea

Step 8: Act

Comments (220)

Test Your Problem-Solving Skills

Find a Neuroscientist

Image of the Week

SUPPORTING PARTNERS

How Good Is Your Problem Solving?

How Good Are You at Solving Problems?

Discover more content

Get 20% off your first year of Mind Tools

Discover something new today

The Life Career Rainbow

How Emotionally Intelligent Are You?

What's Your Leadership Style?