rating scale for problem solving skills

Problem Solving Skills: Performance Review Examples (Rating 1 – 5)

By Status.net Editorial Team on July 21, 2023 — 4 minutes to read

Problem solving is an important skill in any work environment: it includes the ability to identify, understand, and develop solutions to complex issues while maintaining a focus on the end goal. Evaluating this skill in employees during performance reviews can be highly beneficial for both the employee and the organization.

Questions that can help you determine an employee’s rating for problem solving skills:

• How well does the employee define the problem and identify its root cause?
• How creative is the employee in generating potential solutions?
• How effective is the employee in implementing the chosen solution?
• How well does the employee evaluate the effectiveness of the solution and adjust it if necessary?

Related: Best Performance Review Examples for 48 Key Skills

2000+ Performance Review Phrases: The Complete List (Performance Feedback Examples)

Performance Review Phrases and Paragraphs Examples For Problem Solving

5 – outstanding.

Phrases examples:

• Consistently demonstrates exceptional problem-solving abilities
• Proactively identifies issues and offers innovative solutions
• Quickly adapts to unforeseen challenges and finds effective resolutions
• Exceptional problem-solving ability, consistently providing innovative solutions
• Regularly goes above and beyond to find creative solutions to complicated issues
• Demonstrates a keen understanding of complex problems and quickly identifies effective solutions

Paragraph Example 1

“Jane consistently demonstrates outstanding problem-solving skills. She proactively identifies issues in our department and offers innovative solutions that have improved processes and productivity. Her ability to quickly adapt to unforeseen challenges and find effective resolutions is commendable and has proven invaluable to the team.”

Paragraph Example 2

“Sarah has demonstrated an outstanding ability in problem solving throughout the year. Her innovative solutions have significantly improved our department’s efficiency, and she consistently goes above and beyond expectations to find creative approaches to complicated issues.”

4 – Exceeds Expectations

• Demonstrates a strong aptitude for solving complex problems
• Often takes initiative in identifying and resolving issues
• Effectively considers multiple perspectives and approaches before making decisions
• Displayed a consistently strong ability to tackle challenging problems efficiently
• Often takes the initiative to solve problems before they escalate
• Demonstrates a high level of critical thinking when resolving issues

“John exceeds expectations in problem-solving. He has a strong aptitude for solving complex problems and often takes initiative in identifying and resolving issues. His ability to consider multiple perspectives and approaches before making decisions has led to valuable improvements within the team.”

“Sam consistently exceeded expectations in problem solving this year. His efficient handling of challenging issues has made a positive impact on our team, and he often takes the initiative to resolve problems before they escalate. Sam’s critical thinking ability has been a valuable asset to our organization, and we appreciate his efforts.”

3 – Meets Expectations

• Displays adequate problem-solving skills when faced with challenges
• Generally able to identify issues and propose viable solutions
• Seeks assistance when necessary to resolve difficult situations
• Demonstrates a solid understanding of problem-solving techniques
• Capable of resolving everyday issues independently
• Shows perseverance when facing difficult challenges

“Mary meets expectations in her problem-solving abilities. She displays adequate skills when faced with challenges and is generally able to identify issues and propose viable solutions. Mary also seeks assistance when necessary to resolve difficult situations, demonstrating her willingness to collaborate and learn.”

“Sarah meets expectations in her problem-solving abilities. She demonstrates a solid understanding of problem-solving techniques and can resolve everyday issues independently. We value her perseverance when facing difficult challenges and encourage her to continue developing these skills.”

2 – Needs Improvement

• Struggles to find effective solutions to problems
• Tends to overlook critical details when evaluating situations
• Reluctant to seek help or collaborate with others to resolve issues
• Struggles to find effective solutions when faced with complex issues
• Often relies on assistance from others to resolve problems
• May lack confidence in decision-making when solving problems

“Tom’s problem-solving skills need improvement. He struggles to find effective solutions to problems and tends to overlook critical details when evaluating situations. Tom should work on being more willing to seek help and collaborate with others to resolve issues, which will ultimately strengthen his problem-solving abilities.”

“Mark’s problem-solving skills need improvement. He often struggles to find effective solutions for complex issues and seeks assistance from others to resolve problems. We encourage Mark to build his confidence in decision-making and focus on developing his problem-solving abilities.”

1 – Unacceptable

• Fails to identify and resolve problems in a timely manner
• Lacks critical thinking skills necessary for effective problem-solving
• Often creates additional issues when attempting to resolve problems
• Demonstrates a consistent inability to resolve even basic issues
• Often avoids responsibility for problem-solving tasks
• Fails to analyze problems effectively, leading to poor decision-making

“Sally’s problem-solving skills are unacceptable. She consistently fails to identify and resolve problems in a timely manner, and her lack of critical thinking skills hinders her ability to effectively solve challenges. Additionally, her attempts to resolve problems often create additional issues, resulting in a negative impact on the team’s overall performance.”

“Susan’s problem-solving performance has been unacceptable this year. She consistently demonstrates an inability to resolve basic issues and avoids taking responsibility for problem-solving tasks. Her ineffectiveness in analyzing problems has led to poor decision-making. It is crucial that Susan improve her problem-solving skills to succeed in her role.”

• Job Knowledge Performance Review Phrases (Examples)
• Cooperation Skills: Performance Review Examples (Rating 1 - 5)
• 100 Performance Review Phrases for Job Knowledge, Judgment, Listening Skills
• What Are Analytical Skills? [Examples]
• Collaboration Skills: Performance Review Examples (Rating 1 - 5)
• Critical Thinking: Performance Review Examples (Rating 1 - 5)

An IERI – International Educational Research Institute Journal

• Open access
• Published: 10 December 2014

The acquisition of problem solving competence: evidence from 41 countries that math and science education matters

• Ronny Scherer 1 , 2 &
• Jens F Beckmann 3  

Large-scale Assessments in Education volume  2 , Article number:  10 ( 2014 ) Cite this article

14k Accesses

16 Citations

1 Altmetric

Metrics details

On the basis of a ‘problem solving as an educational outcome’ point of view, we analyse the contribution of math and science competence to analytical problem-solving competence and link the acquisition of problem solving competence to the coherence between math and science education. We propose the concept of math-science coherence and explore whether society-, curriculum-, and school-related factors confound with its relation to problem solving.

By using the PISA 2003 data set of 41 countries, we apply multilevel regression and confounder analyses to investigate these effects for each country.

Our results show that (1) math and science competence significantly contribute to problem solving across countries; (2) math-science coherence is significantly related to problem solving competence; (3) country-specific characteristics confound this relation; (4) math-science coherence is linked to capability under-utilisation based on science performance but less on math performance.

Conclusions

In sum, low problem solving scores seem a result of an impeded transfer of subjectspecific knowledge and skills (i.e., under-utilisation of science capabilities in the acquisition of problem solving competence), which is characterised by low levels of math-science coherence.

The ability to solve real-world problems and to transfer problem-solving strategies from domain-specific to domain-general contexts and vice versa has been regarded an important competence students should develop during their education in school (Greiff et al. [ 2013 ]; van Merriënboer [ 2013 ]). In the context of large-scale assessments such as the PISA study problem solving competence is defined as the ability to solve cross-disciplinary and real-world problems by applying cognitive skills such as reasoning and logical thinking (Jonassen [ 2011 ]; OECD [ 2004 ]). Since this competence is regarded a desirable educational outcome, especially math and science educators have focused on developing students’ problem solving and reasoning competence in their respective domain-specific contexts (e.g., Kind [ 2013 ]; Kuo et al. [ 2013 ]; Wu and Adams [ 2006 ]). Accordingly, different conceptual frameworks were proposed that describe the cognitive processes of problem solving such as understanding the problem, building adequate representations of the problem, developing hypotheses, conducting experiments, and evaluating the solution (Jonassen [ 2011 ]; OECD [ 2005 ]). In comparing these approaches in math and science, it seems apparent that there is a conceptual overlap between the problem solving models in these two domains. This overlap triggers the question regarding its contribution to the development of students’ cross-curricular problem-solving competence (Abd-El-Khalick et al. [ 2004 ]; Bassok and Holyoak [ 1993 ]; Hiebert et al. [ 1996 ]).

The operationalization and scaling of performance in PISA assessments enables direct contrasting of scores in students’ competences in math and problem solving. Leutner et al. ([ 2012 ]) suggest that discrepancies between math and problem solving scores are indicative of the relative effectiveness of math education (OECD [ 2004 ]). In line with a “Capability-Utilisation Hypothesis”, it is assumed that math scores that negatively deviate from their problem solving counterpart signify an under-utilisation of students’ problem-solving capabilities as indicated by their scores in generic problem solving.

We intend to extend this view in two ways: First, by introducing the concept of math-science coherence we draw the focus on the potential synergistic link between math and science education and its contribution to the acquisition of problem solving competence. Second, the introduction of a Capability Under-Utilisation Index will enable us to extend the focus of the Capability-Utilisation Hypothesis to both, math and science education. The combination of the concept of math-science coherence with the notion of capability-utilisation will help to further explore the facilitating processes involved in the transition of subject-specific knowledge and skills to the acquisition of problem solving competence. These insights are expected to contribute to a better understanding of meaningful strategies to improve and optimize educational systems in different countries.

Theoretical framework

Problem solving as an educational goal.

In the PISA 2003 framework, problem solving is referred to as “an individual’s capacity to use cognitive processes to resolve real, cross-disciplinary situations where the solution path is not immediately obvious” (OECD [ 2004 ], p. 156). This definition is based on the assumption of domain-general skills and strategies that can be employed in various situations and contexts. These skills and strategies involve cognitive processes such as: Understanding and characterizing the problem, representing the problem, solving the problem, reflecting and communicating the problem solution (OECD [ 2003 ]). Problem solving is often regarded a process rather than an educational outcome, particularly in research on the assessment and instruction of problem solving (e.g., Greiff et al. [ 2013 ]; Jonassen [ 2011 ]). This understanding of the construct is based on the assumption that problem solvers need to perform an adaptive sequence of cognitive steps in order to solve a specific problem (Jonassen [ 2011 ]). Although problem solving has also been regarded as a process skill in large-scale assessments such as the PISA 2003 study, these assessments mainly focus on problem solving performance as an outcome that can be used for international comparisons (OECD [ 2004 ]). However, problem solving competence was operationalized as a construct comprised of cognitive processes. In the context of the PISA 2003 study, these processes were referred to as analytical problem solving, which was assessed by static tasks presented in paper-and-pencil format. Analytical problem-solving competence is related to school achievement and the development of higher-order thinking skills (e.g., Baumert et al. [ 2009 ]; OECD [ 2004 ]; Zohar [ 2013 ]). Accordingly, teachers and educators have focused on models of problem solving as guidelines for structuring inquiry-based processes in their subject lessons (Oser and Baeriswyl [ 2001 ]). Van Merriënboer ([ 2013 ]) pointed out that problem solving should not only be regarded a mere instructional method but also as a major educational goal. Recent curricular reforms have therefore shifted towards the development of problem solving abilities in school (Gallagher et al. [ 2012 ]; Koeppen et al. [ 2008 ]). These reforms were coupled with attempts to strengthen the development of transferable skills that can be applied in real-life contexts (Pellegrino and Hilton [ 2012 ]). For instance, in the context of 21 st century skills, researchers and policy makers have agreed on putting emphasis on fostering skills such as critical thinking, digital competence, and problem solving (e.g., Griffin et al. [ 2012 ]). In light of the growing importance of lifelong learning and the increased complexity of work- and real-life problem situations, these skills are now regarded as essential (Griffin et al. [ 2012 ]; OECD [ 2004 ]). Hence, large-scale educational studies such as PISA have shifted towards the assessment and evaluation of problem solving competence as a 21 st century skill.

The PISA frameworks of math and science competence

In large-scale assessments such as the PISA studies, students’ achievement in the domains of science and mathematics play an important role. Moreover, scientific and mathematical literacy are now regarded essential to being a reflective citizen (Bybee [ 2004 ]; OECD [ 2003 ]). Generally, Baumert et al. ([ 2009 ]) have shown that students’ math and science achievements are highly related to domain-general ability constructs such as reasoning or intelligence. In this context, student achievement refers to “the result of domain-specific processes of knowledge acquisition and information processing” (cf. Baumert et al. [ 2009 ], p. 169). This line of argument is reflected in definitions and frameworks of scientific and mathematical literacy, which are conceptualized as domain-specific competences that are hierarchically organized and build upon abilities closely related to problem solving (Brunner et al. [ 2013 ]).

Scientific literacy has been defined within a multidimensional framework, differentiating between three main cognitive processes, namely describing, explaining, and predicting scientific phenomena, understanding scientific investigations, and interpreting scientific evidence and conclusions (OECD [ 2003 ]). In addition, various types of knowledge such as ‘knowledge about the nature of science’ are considered as factors influencing students’ achievements in this domain (Kind [ 2013 ]). We conclude that the concept of scientific literacy encompasses domain-general problem-solving processes, elements of scientific inquiry (Abd-El-Khalick et al. [ 2004 ]; Nentwig et al. [ 2009 ]), and domain-specific knowledge.

The definition of mathematical literacy refers to students’ competence to utilise mathematical modelling and mathematics in problem-solving situations (OECD [ 2003 ]). Here, we can also identify overlaps between cognitive processes involved in mathematical problem solving and problem solving in general: Structuring, mathematizing, processing, interpreting, and validating (Baumert et al. [ 2009 ]; Hiebert et al. [ 1996 ]; Kuo et al. [ 2013 ]; Polya [ 1945 ]). In short, mathematical literacy goes beyond computational skills (Hickendorff [ 2013 ]; Wu and Adams [ 2006 ]) and is conceptually linked to problem solving.

In the PISA 2003 framework, the three constructs of math, science, and problem solving competence overlap conceptually. For instance, solving the math items requires reasoning, which comprises analytical skills and information processing. Given the different dimensions of the scientific literacy framework, the abilities involved in solving the science items are also related to problem solving, since they refer to the application of knowledge and the performance of inquiry processes (OECD [ 2003 ]). This conceptual overlap is empirically supported by high correlations between math and problem solving ( r  = .89) and between science and problem solving ( r  = .80) obtained for the sample of 41 countries involved in PISA 2003 (OECD [ 2004 ]). The relation between math and science competence was also high ( r  = .83). On the one hand, the sizes of the inter-relationships, give rise to the question regarding the uniqueness of each of the competence measures. On the other hand, the high correlations indicate that problem-solving skills are relevant in math and science (Martin et al. [ 2012 ]). Although Baumert et al. ([ 2009 ]) suggest that the domain-specific competences in math and science require skills beyond problem solving (e.g., the application of domain-specific knowledge) we argue from an assessment perspective that the PISA 2003 tests in math, science, and problem solving measure predominantly basic academic skills relatively independent from academic knowledge (see also Bulle [ 2011 ]).

The concept of capability-utilisation

Discrepancies between students’ performance in math/science and problem solving were studied at country level (OECD [ 2004 ]) and were, for example for math and problem solving scores, interpreted in two ways: (1) If students’ perform better in math than in problem solving, they would “have a better grasp of mathematics content […] after accounting for the level of generic problem-solving skills…” (OECD [ 2004 ], p. 55); (2) If students’ estimated problem-solving competence is higher than their estimated math competence, “… this may suggest that students have the potential to achieve better results in mathematics than that reflected in their current performance…” (OECD [ 2004 ], p. 55). Whilst the latter discrepancy constitutes a capability under-utilisation in math, the former suggests challenges in utilising knowledge and skills acquired in domain-specific contexts in domain-unspecific contexts (i.e., transfer problem).

To quantify the degree to which students are able to transfer their problem solving capabilities from domain-specific problems in math or science to cross-curricular problems, we introduce the Capability Under-Utilisation Index (CUUI) as the relative difference between math or science and problem solving scores:

A positive CUUI indicates that the subject-specific education (i.e., math or science) in a country tends to under-utilise its students’ capabilities to problem solve. A negative CUUI indicates that a country’s educational system fails to fully utilise its students’ capabilities to acquire math and science literacy in the development of problem solving. The CUUI reflects the relative discrepancies between the achievement scores in different domains a .

The concept of math-science coherence

In light of the conceptual and empirical discussion on the relationship between math, science, and problem solving competence, we introduce the concept of math-science coherence as follows: First, math-science coherence refers to the set of cognitive processes involved in both subjects and thus represents processes which are related to reasoning and information processing, relatively independent from domain-specific knowledge. Second, math-science coherence reflects the degree to which math and science education is harmonized as a feature of the educational environment in a country. This interpretation is based on the premise that PISA measures students’ competence as educational outcomes (OECD [ 2004 ]). The operationalization of math-science coherence is realized by means of the correlation between math and science scores [ r (M,S)] at the country level. Low math-science coherence indicates that students who are successful in the acquisition of knowledge and skills in math are not necessarily successful in the acquisition of knowledge and skills in science and vice versa.

On the basis of this conceptualization of math-science coherence, we expect a significant and positive relation to problem solving scores, since the conceptual overlap between mathematical and scientific literacy refers to cognitive abilities such as reasoning and information processing that are also required in problem solving (Arts et al. [ 2006 ]; Beckmann [ 2001 ]; Wüstenberg et al. [ 2012 ]). Hence, we assert that math-science coherence facilitates the transfer of knowledge, skills, and insights across subjects resulting in better problem solving performance (OECD [ 2004 ]; Pellegrino and Hilton [ 2012 ]).

We also assume that math-science coherence as well as capability utilisation is linked to characteristics of the educational system of a country. For instance, as Janssen and Geiser ([ 2012 ]) and Blömeke et al. ([ 2011 ]) suggested, the developmental status of a country, measured by the Human Development Index (HDI; UNDP [ 2005 ]), is positively related to students’ academic achievements as well as to teachers’ quality of teaching. Furthermore, the socio-economic status of a country co-determines characteristics of its educational system, which ultimately affects a construct referred to as national intelligence (Lynn and Meisenberg [ 2010 ]). Research also indicated that curricular settings and educational objectives are related to school achievement in general (Bulle [ 2011 ]; Martin et al. [ 2004 ]). Besides these factors, school- and classroom-related characteristics might also confound the relation between math-science coherence and problem solving. For instance, the schools’ autonomy in developing curricula and managing educational resources might facilitate the incorporation of inquiry- and problem-based activities in science lessons (Chiu and Chow [ 2011 ]). These factors have been discussed as being influential to students’ competence development (OECD [ 2004 ], [ 2005 ]). Ewell ([ 2012 ]) implies that cross-national differences in problem solving competence might be related to differences in education and in using appropriate teaching material. These factors potentially confound the relation between math-science coherence and problem solving.

Discrepancies between math and problem solving scores are discussed in relation to quality of education. Although research has found that crossing the borders between STEM subjects positively affects students’ STEM competences (e.g., National Research Council NRC [ 2011 ]), we argue that the PISA analyses have fallen short in explaining cross-country differences in the development of problem solving competence, since they ignored the link between math and science competences and the synergistic effect of learning universally applicable problem-solving skills in diverse subject areas. Hence, we use the concept of math-science coherence to provide a more detailed description of the discrepancies between problem solving and domain-specific competences. In this regard, we argue that the coherence concept indicates the synergistic potential and students’ problem-solving competence the success of transfer.

The present study

The current study is based on the premise that in contrast to math and science competence problem solving competence is not explicitly taught as a subject at school. Problem solving competence, however, is an expected outcome of education (van Merriënboer [ 2013 ]). With the first step in our analyses, we seek to establish whether math and science education are in fact main contributors to the acquisition of problem solving competence. On the basis of this regression hypothesis, we subsequently focus on the question whether there are significant and systematic differences between countries ( Moderation-Hypothesis ). In light of the conceptual overlap due to cognitive processes involved in dealing with math, science and problem solving tasks and the shared item format employed in the assessments, we expect math and science competence scores to substantially predict scores in problem solving competence. Furthermore, since math and science education are differently organized across the 41 countries participating in the PISA 2003 study, differences in the contribution are also expected.

On the basis of these premises, we introduce the concept of math-science coherence, operationalised as the correlation between math and science scores [ r (M,S)], and analyse its relationship to problem solving and the effects of confounders (i.e., country characteristics) as a step of validation. Since math, science, and problem solving competence show a conceptual overlap, we expect problem solving and math-science coherence to be positively related. Countries’ educational systems differ in numerous aspects, their educational structure, and their educational objectives. Countries also differ with regard to the frequency of assessments, the autonomy of schools in setting up curricula and resources, and the educational resources available. Consequently, we expect the relation between math-science coherence and problem solving competence to be confounded by society-, curriculum-, and school-related factors ( Confounding-Hypothesis ).

In a final step, we aim to better understand the mechanisms with which math and science education contributes to the acquisition of problem-solving competence by exploring how math-science coherence, capability utilisation, and problem solving competence are related. We thus provide new insights into factors related to the transfer between students’ domain-specific and cross-curricular knowledge and skills ( Capability-Utilisation Hypothesis ).

In PISA 2003, a total sample of N  = 276,165 students (49.4% female) from 41 countries participated. The entire sample was randomly selected by applying a two-step sampling procedure: First, schools were chosen within a country. Second, students were chosen within these schools. This procedure consequently led to a clustered structure of the data set, as students were nested in 10,175 schools. On average, 27 students per school were chosen across schools within countries. Students’ mean age was 15.80 years ( SD  = 0.29 years) ranging from 15.17 to 16.25 years.

In the PISA 2003 study, different assessments were used in order to measure students’ competence in math, science, and problem solving. These assessments were administered as paper-and-pencil tests within a multi-matrix design (OECD [ 2005 ]). In this section, the assessments and further constructs are described that served as predictors of the contribution of math and science competence to problem solving at the country level.

Student achievement in math, science, and problem solving

In order to assess students’ competence to solve cross-curricular problems (i.e., analytical problem solving requiring information retrieval and reasoning), students had to work on an analytical problem-solving test. This test comprised a total of 19 items (7 items referred to trouble-shooting, 7 items referred to decision-making, and 5 items referred to system analysis and design; see OECD [ 2004 ]). Items were coded according to the PISA coding scheme, resulting in dichotomous and polytomous scores (OECD [ 2005 ]). Based on these scores, models of item response theory were specified in order to obtain person and item parameters (Leutner et al. [ 2012 ]). The resulting plausible values could be regarded as valid indicators of students’ abilities in problem solving (Wu [ 2005 ]). The problem solving test showed sufficient reliabilities between .71 and .89 for the 41 countries.

To assess mathematical literacy as an indicator of math competence , an 85-items test was administered (for details, refer to OECD [ 2003 ]). Responses were dichotomously or polytomously scored. Again, plausible values were obtained as person ability estimates and reliabilities were good (range: 0.83 – 0.93). In the context of mathematical literacy, students were asked to solve real-world problems by applying appropriate mathematical models. They were prompted to “identify and understand the role mathematics plays in the world, to make well-founded judgements and to use […] mathematics […]” (OECD [ 2003 ], p. 24).

Scientific literacy as a proxy for science competence was assessed by using problems referring to different content areas of science in life, health, and technology. The reliability estimates for the 35 items in this test ranged between .68 and .88. Again, plausible values served as indicators of this competence.

Country-specific characteristics

In our analyses, we incorporated a range of country-specific characteristics that can be subdivided into three main categories. These are: society-related factors, curriculum-related factors, and school-related factors. Country-specific estimates of National Intelligence as derived by Lynn and Meisenberg ([ 2010 ]) as well as the Human Development Index (HDI) were subsumed under society-related factors . The HDI incorporates indicators of a country’s health, education, and living standards (UNDP [ 2005 ]). Both variables are conceptualised as factors that contribute to country-specific differences in academic performance.

Holliday and Holliday ([ 2003 ]) emphasised the role of curricular differences in the understanding of between-country variance in test scores. We incorporated two curriculum-related factors in our analyses. First, we used Bulle’s ([ 2011 ]) classification of curricula into ‘progressive’ and ‘academic’. Bulle ([ 2011 ]) proposed this framework and classified the PISA 2003 countries according to their educational model. In her framework, she distinguishes between ‘academic models’ which are primarily geared towards teaching academic subjects (e.g., Latin, Germanic, and East-Asian countries) and ‘progressive models’ which focus on teaching students’ general competence in diverse contexts (e.g., Anglo-Saxon and Northern countries). In this regard, academic skills refer to the abilities of solving academic-type problems, whereas so called progressive skills are needed in solving real-life problems (Bulle [ 2011 ]). It can be assumed that educational systems that focus on fostering real-life and domain-general competence might be more conducive to successfully tackling the kind of problem solving tasks used in PISA (Kind [ 2013 ]). This classification of educational systems should be seen as the two extreme poles of a continuum rather than as a strict dichotomy. In line with the reflections above, we would argue that academic and progressive skills are not exclusively distinct, since both skills utilise sets of cognitive processes that largely overlap (Klahr and Dunbar [ 1988 ]). The fact that curricular objectives in some countries are shifting (e.g., in Eastern Asia) makes a clear distinction between both models even more difficult. Nonetheless, we will use this form of country-specific categorization based on Bulle’s model in our analyses.

Second, we considered whether countries’ science curricula were ‘integrated’ or ‘not integrated’ (Martin et al. [ 2004 ]). In this context, integration refers to linking multiple science subjects (biology, chemistry, earth science, and physics) to a unifying theme or issue (cf. Drake and Reid [ 2010 ], p. 1).

In terms of school-related factors, we used the PISA 2003 scales of ‘Frequency of assessments in schools’, ‘Schools’ educational resources’, and ‘School autonomy towards resources and curricula’ from the school questionnaire. Based on frequency and rating scales, weighted maximum likelihood estimates (WLE) indicated the degree to which schools performed in these scales (OECD [ 2005 ]).

The country-specific characteristics are summarized in the Table 1 .

The PISA 2003 assessments utilised a randomized incomplete block design to select different test booklets which covered the different content areas of math, science, and problem solving (Brunner et al. [ 2013 ]; OECD [ 2005 ]). The test administration took 120 minutes, and was managed for each participating country separately. It was established that quality standards of the assessment procedure were high.

Statistical analyses

In PISA 2003, different methods of obtaining person estimates with precise standard errors were applied. The most accurate procedure produced five plausible values, which were drawn from a person ability distribution (OECD [ 2005 ]). To avoid missing values in these parameters and to obtain accurate estimates, further background variables were used within the algorithms (Wu [ 2005 ]). The resulting plausible values were subsequently used as indicators of students’ competence in math, science, and problem solving. By applying Rubin’s combination rules (Bouhilila and Sellaouti [ 2013 ]; Enders [ 2010 ]), analyses were replicated with each of the five plausible values and then combined. In this multiple imputation procedure, standard errors were decomposed to the variability across and within the five imputations (Enders [ 2010 ]; OECD [ 2005 ]; Wu [ 2005 ]).

Within the multilevel regression analyses for each country, we specified the student level as level 1 and the school level as level 2. Since PISA 2003 applied a random sampling procedure at the student and the school level, we decided to control for the clustering of data at these two levels (OECD [ 2005 ]). In addition to this two-level procedure, we regarded the 41 countries as multiple groups (fixed effects). This decision was based on our assumption that the countries selected in PISA 2003 did not necessarily represent a sample of a larger population (Martin et al. [ 2012 ]). Moreover, we did not regard the effects of countries as interchangeable, because, given the specific characteristics of education and instruction within countries; we argue that the effects of competences in mathematics and science on problem solving have their own distinct interpretation in each country (Snijders and Bosker [ 2012 ]). The resulting models were compared by taking into account the Akaike’s information criteria ( AIC ), Bayesian information criteria ( BIC ), and the sample-size adjusted BIC . Also, a likelihood ratio test of the log-Likelihood values ( LogL ) was applied (Hox [ 2010 ]).

To test the Moderation-Hypothesis, we first specified a two-level regression model with problem solving scores as outcomes at the student level (level 1), which allowed variance in achievement scores across schools (level 2). In this model, math and science scores predicted problem solving scores at the student level. To account for differences in the probabilities of being selected as a student within the 41 countries and to adjust the standard errors of regression parameters, we used the robust maximum likelihood (MLR) estimator and students’ final weights (see also Brunner et al. [ 2013 ]; OECD [ 2005 ]). All analyses were conducted in Mplus 6.0 by using the TYPE = IMPUTATION option (Muthén and Muthén [ 2010 ]). As Hox ([ 2010 ]) suggested, using multilevel regression models without taking into account the clustering of data in schools often leads to biased estimates, since achievement variables often have substantial variance at the school level. Consequently, we allowed for level-2-variance within the scores.

After having established whether success in math and science education contributes to the development in problem solving competence across the 41 countries, we then tested whether cross-country differences in the unstandardized regression coefficients were statistically significant by using a multi-group regression model, in which the coefficients were constrained to be equal across countries. We compared this model with the freely estimated model.

Finally, we conceptualized the correlation between math and science scores as an indicator of the level of coherence in math and science education in a country. In relation to the Confounding-Hypothesis, we tested country-specific characteristics for their potentially confounding effects on the relation between math-science coherence and problem solving competence. Following the recommendations proposed by (MacKinnon et al. [ 2000 ]), the confounding analysis was conducted in two steps: (1) we estimated two regression equations. In the first equation, problem solving scores across the 41 countries were regressed on math-science coherence. In the second equation, the respective country characteristics were added as further predictors; (2) the difference between the regression coefficients for math-science coherence obtained in either equation represented the magnitude of a potential confounder effect.

Lastly, we tested the Capability-Utilisation Hypothesis by investigating the bivariate correlations among the CUU Indices and math-science coherence.

Regressing problem solving on math and science performance

To test the Moderation-Hypothesis, we specified regression models with students’ problem-solving score as the outcome and math and science scores as predictors for each of the 41 countries. Due to the clustering of data in schools, these models allowed for between-level variance. Intraclass correlations (ICC-1) for math, science, and problem solving performance ranged between .03 and .61 for the school level ( M  = .33, SD  = .16).

We specified multilevel regression models for each country separately. These results are reported in Table  2 . The regression coefficients for math on problem solving ranged from .53 to .82 with an average of M( β Math )  = .67 ( SD  = .06). The average contribution of science towards problem solving was M( β Science )  = .16 ( SD  = .09, Min  = -.06, Max  = .30). The combination of the distributions of both parameters resulted in substantial differences in the variance explanations of the problem solving scores across the 41 countries ( M[R 2 ]  = .65, SD  = .15, Min  = .27, Max  = .86). To test whether these differences were statistically significant, we constrained the regression coefficients of math and science competence within the multi-group regression model to be equal across the 41 countries. Compared to the freely estimated model ( LogL  = -4,561,273.3, df  = 492, AIC  = 9,123,530.5, BIC  = 9,128,410.7), the restricted model was empirically not preferred LogL  = -4,564,877.9, df  = 412, AIC  = 9,130,579.8, BIC  = 9,134,917.6; Δχ 2 [80] = 7,209.2, p  < .001. These findings lend evidence for the Moderation-Hypothesis.

From a slightly different perspective, the country-specific amount of variance in problem solving scores that is explained by the variation in math and science performance scores ( R 2 ) is strongly associated with the country’s problem solving score ( r  = .77, p  < .001), which suggests that the contribution of science and math competence to the acquisition of problem solving competence was significantly lower in low-performing countries.

As shown in Table  2 , the regression weights of math and science were significant for all but two countries. Across countries the regression weight for math tended to be higher than the regression weight for science when predicting problem solving competence. This finding indicates a stronger overlap between students’ competences in mathematics and problem solving on the one hand and similarities between the assessments in both domains on the other hand.

Validating the concept of math-science coherence

In order to validate the concept of math-science coherence, which is operationalised as the correlation between math and science scores [ r (M,S)], we explored its relation to problem solving and country characteristics.

Regarding the regression outcomes shown in Table  2 , it is apparent that math-science coherence varied considerably across countries, ranging from .39 to .88 with an average of M(r)  = .70 ( SD  = .13). Interestingly, countries’ level of coherence in math-science education was substantially related to their problem solving scores ( r  = .76, p  < .001). An inspection of Figure  1 reveals that this effect was mainly due to countries that both achieve low problem solving scores and show relatively low levels of math-science coherence (see bottom left quadrant in Figure  1 ), whilst amongst the remaining countries the correlational link between math-science coherence and problem solving score was almost zero ( r  = -.08, p  = .71) b . This pattern extends the moderation perspective on the presumed dependency of problem solving competence from math and science competences.

The relation between math-science coherence and problem solving performance across the 41 countries.

As a result of the moderator analysis, we know that countries not only differ in regard to their average problem-solving scores and level of coherence between math and science, countries also differ in the strengths with which math-science coherence predicts problem solving scores. To better understand the conceptual nature of the link between math-science coherence and problem solving, we now attempt to adjust this relationship for potential confounding effects that country-specific characteristics might have. To this end, we employed linear regression and path analysis with students’ problem-solving scores as outcomes, math-science coherence (i.e., r [M,S]) as predictor, and country characteristics as potential confounders.

To establish whether any of the country characteristics had a confounding effect on the link between math-science coherence and problem solving competence, two criteria had to be met: (1) a reduction of the direct effect of math-science coherence on problem solving scores, and (2) testing the difference between the direct effect within the baseline Model M0 and the effect with the confounding Model M1 (Table  3 ).

Regarding the society-related factors, both the countries’ HDI and their national intelligence were confounders with a positive effect. Furthermore, the countries’ integration of the science curriculum was also positively related to the problem solving performance. Finally, the degree of schools’ autonomy towards educational resources and the implementation of curricula and the frequency of assessments were school-related confounders, the former with a positive effect whilst the latter represents a negative confounder. The direct effect of math-science coherence to problem solving decreased and thus indicated that confounding was present (MacKinnon et al. [ 2000 ]).

These findings provide evidence on the Confounding-Hypothesis and support our expectations on the relation between math-science coherence, problem solving, and country characteristics. We regard these results as evidence for the validity of the math-science coherence measure.

Relating math-science coherence to the capability under-utilisation indices

To advance our understanding of the link between math-science coherence and problem solving scores, we tested the Capability-Utilisation Hypothesis. To this end, we explored the relationship between math-science coherence and the CUU Indices for math and science, respectively. For math competence the average Capability Under-Utilisation Index was rather neutral with M CUUI-Math  = -0.001 ( SD  = 0.02). This suggests that, on average, all countries sufficiently utilise their students’ math capabilities in facilitating the development of problem solving competence (i.e., transfer). It also suggests that math education across participating countries tends to sufficiently utilise generic problem-solving skills (Figure  2 ). The picture is different for science education. Here, the Capability Under-Utilisation Indices and their variation across the participating countries ( M CUUI-Science  = -0.01, SD  = 0.04) suggest that in a range of countries knowledge and skills taught in science education tend to be under-utilised in the facilitation of the acquisition of problem solving competence (Figure  3 ).

The relation between math-science coherence and the capability under-utilisation index for math and problem solving scores across the 41 countries.

The relation between math-science coherence and the capability under-utilisation index for science and problem solving scores across the 41 countries.

For math competence, the relative difference to problem solving was not related to math-science coherence ( r  = .02, p  = .89; Figure  2 ). In contrast, the Capability Under-Utilisation Index for science showed a strong positive correlation with math-science coherence ( r  = .76, p  < .001; Figure  3 ), indicating that low levels of coherence between math and science education were associated with a less effective transfer of domain-specific knowledge and skills to problem solving.

The present study was aimed at investigating the differences in the contribution of math and science competence to problem solving competence across the 41 countries that participated in the PISA 2003 study (Moderation-Hypothesis). To this end, we proposed the concept of math-science coherence and explored its relationship to problem solving competence and how this relationship is confounded by country characteristics (Confounding-Hypothesis). To further extend our understanding of the link between math-science coherence and problem solving, we introduced the concept of capability-utilisation. Testing the Capability-Utilisation Hypothesis enabled us to identify what may contribute to varying levels of math-science coherence and ultimately the development of problem solving competence.

The contribution of math and science competence across countries

Regarding the prediction of problem solving competence, we found that in most countries, math and science competence significantly contributed to students’ performance in analytical problem solving. This finding was expected based on the conceptualizations of mathematical and scientific literacy within the PISA framework referring to shared cognitive processes such as information processing and reasoning (Kind [ 2013 ]; OECD [ 2005 ]), which are regarded as components of problem solving (Bybee [ 2004 ]; Klahr and Dunbar [ 1988 ]; Mayer [ 2010 ]).

It is noteworthy that, for some of the below-average performing countries, science competence did not significantly contribute to the prediction of problem solving competence. It can be speculated that education in these countries is more geared towards math education and modelling processes in mathematical scenarios, whilst the aspect of problem solving in science is less emphasised (Janssen and Geiser [ 2012 ]). The results of multilevel regression analyses supported this interpretation by showing that math competence was a stronger predictor of problem solving competence. On the one hand, this finding could be due to the design of the PISA tests (Adams [ 2005 ]), since math and problem solving items are designed in such a way that modelling real-life problems is required, whereas science items are mostly domain-specific and linked to science knowledge (Nentwig et al. [ 2009 ]; OECD [ 2004 ]). Moreover, one may argue that math and problem solving items allow students to employ different solution strategies, whereas science items offer fewer degrees of freedom for test takers (Nentwig et al. [ 2009 ]). In particular, the shared format of items in math, science, and problem solving may explain an overlap between their cognitive demands. For instance, most of the items were designed in such a way that students had to extract and identify relevant information from given tables or figures in order to solve specific problems. Hence, these items were static and did not require knowledge generation by interaction or exploration but rather the use of given information in problem situations (Wood et al. [ 2009 ]). In contrast to the domain-specific items in math and science, problem solving items did not require the use of prior knowledge in math and science (OECD [ 2004 ]). In addition, some of the math and science items involved cognitive operations that were specific to these domains. For instance, students had to solve a number of math items by applying arithmetic and combinatorial operations (OECD [ 2005 ]). Finally, since items referred to contextual stimuli, which were presented in textual formats, reading ability can be regarded as another, shared demand of solving the items. Furthermore, Rindermann ([ 2007 ]) clearly showed that the shared demands of the achievement tests in large-scale assessments such as PISA were strongly related to students’ general reasoning skills. This finding is in line with the strong relations between math, science, and problem solving competence, found in our study. The interpretation of the overlap between the three competences can also be interpreted from a conceptual point of view. In light of the competence frameworks in PISA, we argue that there are a number of skills that can be found in math, science, and problem solving: information retrieval and processing, knowledge application, and evaluation of results (Griffin et al. [ 2012 ]; OECD [ 2004 ], [ 2005 ]). These skills point out to the importance of reasoning in the three domains (Rindermann [ 2007 ]). Thus, the empirical overlap between math and problem solving can be explained by shared processes of, what Mayer ([ 2010 ]) refers to as, informal reasoning. On the other hand, the stronger effect of math competence could be an effect of the quality of math education. Hiebert et al. ([ 1996 ]) and Kuo et al. ([ 2013 ]) suggested that math education is more based on problem solving skills than other subjects in school (e.g., Polya [ 1945 ]). Science lessons, in contrast, are often not necessarily problem-based, despite the fact that they often start with a set problem. Risch ([ 2010 ]) showed in a cross-national review that science learning was more related to contents and contexts rather than to generic problem-solving skills. These tendencies might lead to a weaker contribution of science education to the development of problem solving competence (Abd-El-Khalick et al. [ 2004 ]).

In sum, we found support on the Moderation-Hypothesis, which assumed systematic differences in the contribution of math and science competence to problem solving competence across the 41 PISA 2003 countries.

The relation to problem solving

In our study, we introduced the concept of math-science coherence, which reflects the degree to which math and science education are harmonized. Since mathematical and scientific literacy show a conceptual overlap, which refers to a set of cognitive processes that are linked to reasoning and information processing (Fensham and Bellocchi [ 2013 ]; Mayer [ 2010 ]), a significant relation between math-science coherence and problem solving was expected. In our analyses, we found a significant and positive effect of math-science coherence on performance scores in problem solving. In this finding we see evidence for the validity of this newly introduced concept of math-science coherence and its focus on the synergistic effect of math and science education on problem solving. The results further suggest that higher levels of coordination between math and science education has beneficial effects on the development of cross-curricular problem-solving competence (as measured within the PISA framework).

Confounding effects of country characteristics

As another step of validating the concept of math-science coherence, we investigated whether country-specific characteristics that are linked to society-, curriculum-, and school-related factors confounded its relation to problem solving. Our results showed that national intelligence, the Human Development Index, the integration of the science curriculum, and schools’ autonomy were positively linked to math-science coherence and problem solving, whilst a schools’ frequency of assessment had a negative confounding effect.

The findings regarding the positive confounders are in line with and also extend a number of studies on cross-country differences in education (e.g., Blömeke et al. [ 2011 ]; Dronkers et al. [ 2014 ]; Janssen and Geiser [ 2012 ]; Risch [ 2010 ]). Ross and Hogaboam-Gray ([ 1998 ]), for instance, found that students benefit from an integrated curriculum, particularly in terms of motivation and the development of their abilities. In the context of our confounder analysis, the integration of the science curriculum as well as the autonomy to allocate resources is expected to positively affect math-science coherence. At the same time, an integrated science curriculum with a coordinated allocation of resources may promote inquiry-based experiments in science courses, which is assumed to be beneficial for the development of problem solving within and across domains. Teaching science as an integrated subject is often regarded a challenge for teachers, particularly when developing conceptual structures in science lessons (Lang and Olson, [ 2000 ]), leading to teaching practices in which cross-curricular competence is rarely taken into account (Mansour [ 2013 ]; van Merriënboer [ 2013 ]).

The negative confounding effect of assessment frequency suggests that high frequencies of assessment, as it presumably applies to both math and science subjects, contribute positively to math-science coherence. However, the intended or unintended engagement in educational activities associated with assessment preparation tends not to be conducive to effectively developing domain-general problem solving competence (see also Neumann et al. [ 2012 ]).

The positive confounder effect of HDI is not surprising as HDI reflects a country’s capability to distribute resources and to enable certain levels of autonomy (Reich et al. [ 2013 ]). To find national intelligence as a positive confounder is also to be expected as the basis for its estimation are often students’ educational outcome measures (e.g., Rindermann [ 2008 ]) and, as discussed earlier, academic achievement measures share the involvement of a set of cognitive processes (Baumert et al. [ 2009 ]; OECD [ 2004 ]).

In summary, the synergistic effect of a coherent math and science education on the development of problem solving competence is substantially linked to characteristics of a country’s educational system with respect to curricula and school organization in the context of its socio-economic capabilities. Math-science coherence, however, also is linked to the extent to which math or science education is able to utilise students’ educational capabilities.

Math-science coherence and capability-utilisation

So far, discrepancies between students’ performance in math and problem solving or science and problem solving have been discussed as indicators of students’ capability utilisation in math or science (Leutner et al. [ 2012 ]; OECD [ 2004 ]). We have extended this perspective by introducing Capability Under-Utilisation Indices for math and science to investigate the effectiveness with which knowledge and skills acquired in the context of math or science education are transferred into cross-curricular problem-solving competence. The Capability Under-Utilisation Indices for math and science reflect a potential quantitative imbalance between math, science, and problem solving performance within a country, whilst the also introduced concept of math-science coherence reflects a potential qualitative imbalance between math and science education.

The results of our analyses suggest that an under-utilisation of problem solving capabilities in the acquisition of science literacy is linked to lower levels of math-science coherence, which ultimately leads to lower scores in problem solving competence. This interpretation finds resonance in Ross and Hogaboam-Gray’s ([ 1998 ]) argumentation for integrating math and science education and supports the attempts of math and science educators to incorporate higher-order thinking skills in teaching STEM subjects (e.g., Gallagher et al. [ 2012 ]; Zohar [ 2013 ]).

In contrast, the CUU Index for math was not related to math-science coherence in our analyses. This might be due to the conceptualizations and assessments of mathematical literacy and problem solving competence. Both constructs share cognitive processes of reasoning and information processing, resulting in quite similar items. Consequently, the transfer from math-related knowledge and skills to cross-curricular problems does not necessarily depend on how math and science education are harmonised, since the conceptual and operational discrepancy between math and problem solving is rather small.

Math and science education do matter to the development of students’ problem-solving skills. This argumentation is based on the assumption that the PISA assessments in math, science, and problem solving are able to measure students’ competence as outcomes, which are directly linked to their education (Bulle [ 2011 ]; Kind [ 2013 ]). In contrast to math and science competence, problem solving competence is not explicitly taught as a subject. Problem solving competence requires the utilisation of knowledge and reasoning skills acquired in specific domains (Pellegrino and Hilton [ 2012 ]). In agreement with Kuhn ([ 2009 ]), we point out that this transfer does not happen automatically but needs to be actively facilitated. In fact, Mayer and Wittrock ([ 2006 ]) stressed that the development of transferable skills such as problem solving competence needs to be fostered within specific domains rather than taught in dedicated, distinct courses. Moreover, they suggested that students should develop a “repertoire of cognitive and metacognitive strategies that can be applied in specific problem-solving situations” (p. 299). Beyond this domain-specific teaching principle, research also proposes to train the transfer of problem solving competence in domains that are closely related (e.g., math and science; Pellegrino and Hilton [ 2012 ]). In light of the effects of aligned curricula (as represented by the concept of math-science coherence), we argue that educational efforts to increase students’ problem solving competence may focus on a coordinated improvement of math and science literacy and fostering problem solving competence within math and science. The emphasis is on coordinated, as the results of our analyses indicated that the coherence between math and science education, as a qualitative characteristic of a country’s educational system, is a strong predictor of problem solving competence. This harmonisation of math and science education may be achieved by better enabling the utilisation of capabilities, especially in science education. Sufficiently high levels of math-science coherence could facilitate the emergence of educational synergisms, which positively affect the development of problem solving competence. In other words, we argue for quantitative changes (i.e., improve science attainment) in order to achieve qualitative changes (i.e., higher levels of curriculum coherence), which are expected to create effective transitions of subject-specific knowledge and skills into subject-unspecific competences to solve real-life problems (Pellegrino and Hilton [ 2012 ]; van Merriënboer [ 2013 ]).

Finally, we encourage research that is concerned with the validation of the proposed indices for different forms of problem solving. In particular, we suggest studying the facilities of the capability-under-utilisation indices for analytical and dynamic problem solving, as assessed in the PISA 2012 study (OECD [ 2014 ]). Due to the different cognitive demands in analytical and dynamic problems (e.g., using existing knowledge vs. generating knowledge; OECD [ 2014 ]), we suspect differences in capability utilisation in math and science. This research could provide further insights into the role of 21 st century skills as educational goals.

a The differences between students’ achievement in mathematics and problem solving, and science and problem solving have to be interpreted relative to the OECD average, since the achievement scales were scaled with a mean of 500 and a standard deviation of 100 for the OECD countries (OECD [ 2004 ], p. 55). Although alternative indices such as country residuals may also be used in cross-country comparisons (e.g., Olsen [ 2005 ]), we decided to use CUU indices, as they reflect the actual differences in achievement scores.

b In addition, we checked whether this result was due to the restricted variances in low-performing countries and found that neither ceiling nor floor effects in the problem solving scores existed. The problem solving scale differentiated sufficiently reliably in the regions below and above the OECD mean of 500.

Abd-El-Khalick F, Boujaoude S, Duschl R, Lederman NG, Mamlok-Naaman R, Hofstein A, Niaz M, Treagust D, Tuan H-L: Inquiry in science education: International perspectives. Science Education 2004, 88: 397–419. doi:10.1002/sce.10118 10.1002/sce.10118

Article   Google Scholar  

Adams RJ: Reliability as a measurement design effect. Studies in Educational Evaluation 2005, 31: 162–172. doi:10.1016/j.stueduc.2005.05.008 10.1016/j.stueduc.2005.05.008

Arts J, Gijselaers W, Boshuizen H: Understanding managerial problem solving, knowledge use and information processing: Investigating stages from school to the workplace. Contemporary Educational Psychology 2006, 31: 387–410. doi:10.1016/j.cedpsych.2006.05.005 10.1016/j.cedpsych.2006.05.005

Bassok M, Holyoak K: Pragmatic knowledge and conceptual structure: determinants of transfer between quantitative domains. In Transfer on trial: intelligence, cognition, and instruction . Edited by: Detterman DK, Sternberg RJ. Ablex, Norwood, NJ; 1993:68–98.

Google Scholar  

Baumert J, Lüdtke O, Trautwein U, Brunner M: Large-scale student assessment studies measure the results of processes of knowledge acquisition: evidence in support of the distinction between intelligence and student achievement. Educational Research Review 2009, 4: 165–176. doi:10.1016/j.edurev.2009.04.002 10.1016/j.edurev.2009.04.002

Beckmann JF: Zur Validierung des Konstrukts des intellektuellen Veränderungspotentials [On the validation of the construct of intellectual change potential] . logos, Berlin; 2001.

Blömeke S, Houang R, Suhl U: TEDS-M: diagnosing teacher knowledge by applying multidimensional item response theory and multiple-group models. IERI Monograph Series: Issues and Methodologies in Large-Scale Assessments 2011, 4: 109–129.

Bouhilila D, Sellaouti F: Multiple imputation using chained equations for missing data in TIMSS: a case study. Large-scale Assessments in Education 2013, 1: 4.

Brunner M, Gogol K, Sonnleitner P, Keller U, Krauss S, Preckel F: Gender differences in the mean level, variability, and profile shape of student achievement: Results from 41 countries. Intelligence 2013, 41: 378–395. doi:10.1016/j.intell.2013.05.009 10.1016/j.intell.2013.05.009

Bulle N: Comparing OECD educational models through the prism of PISA. Comparative Education 2011, 47: 503–521. doi: 10.1080/03050068.2011.555117 10.1080/03050068.2011.555117

Bybee R: Scientific Inquiry and Science Teaching. In Scientific Inquiry and the Nature of Science . Edited by: Flick L, Lederman N. Springer & Kluwers, New York, NY; 2004:1–14. doi:10.1007/978–1-4020–5814–1_1

Chiu M, Chow B: Classroom discipline across forty-One countries: school, economic, and cultural differences. Journal of Cross-Cultural Psychology 2011, 42: 516–533. doi:10.1177/0022022110381115 10.1177/0022022110381115

Drake S, Reid J: Integrated curriculum: Increasing relevance while maintaining accountability. Research into Practice 2010, 28: 1–4.

Dronkers J, Levels M, de Heus M: Migrant pupils’ scientific performance: the influence of educational system features of origin and destination countries. Large-scale Assessments in Education 2014, 2: 3.

Enders C: Applied Missing Data Analysis . The Guilford Press, New York, NY; 2010.

Ewell P: A world of assessment: OECD’s AHELO initiative. Change: The Magazine of Higher Learning 2012, 44: 35–42. doi:10.1080/00091383.2012.706515 10.1080/00091383.2012.706515

Fensham P, Bellocchi A: Higher order thinking in chemistry curriculum and its assessment. Thinking Skills and Creativity 2013, 10: 250–264. doi:10.1016/j.tsc.2013.06.003 10.1016/j.tsc.2013.06.003

Gallagher C, Hipkins R, Zohar A: Positioning thinking within national curriculum and assessment systems: perspectives from Israel, New Zealand and Northern Ireland. Thinking Skills and Creativity 2012, 7: 134–143. doi:10.1016/j.tsc.2012.04.005 10.1016/j.tsc.2012.04.005

Greiff S, Holt D, Funke J: Perspectives on problem solving in educational assessment: analytical, interactive, and collaborative problem solving. The Journal of Problem Solving 2013, 5: 71–91. doi:10.7771/1932–6246.1153 10.7771/1932-6246.1153

Griffin P, Care E, McGaw B: The changing role of education and schools. In Assessment and Teaching of 21st Century Skills . Edited by: Griffin P, McGaw B, Care E. Springer, Dordrecht; 2012:1–15. 10.1007/978-94-007-2324-5_1

Chapter   Google Scholar  

Hickendorff M: The language factor in elementary mathematics assessments: Computational skills and applied problem solving in a multidimensional IRT framework. Applied Measurement in Education 2013, 26: 253–278. doi:10.1080/08957347.2013.824451 10.1080/08957347.2013.824451

Hiebert J, Carpenter T, Fennema E, Fuson K, Human P, Murray H, Olivier A, Wearne D: Problem Solving as a Basis for Reform in Curriculum and Instruction: The Case of Mathematics. Educational Researcher 1996, 25: 12–21. doi:10.3102/0013189X025004012 10.3102/0013189X025004012

Holliday W, Holliday B: Why using international comparative math and science achievement data from TIMSS is not helpful. The Educational Forum 2003, 67: 250–257. 10.1080/00131720309335038

Hox J: Multilevel Analysis . 2nd edition. Routlegde, New York, NY; 2010.

Janssen A, Geiser C: Cross-cultural differences in spatial abilities and solution strategies: An investigation in Cambodia and Germany. Journal of Cross-Cultural Psychology 2012, 43: 533–557. doi:10.1177/0022022111399646 10.1177/0022022111399646

Jonassen D: Learning to solve problems . Routledge, New York, NY; 2011.

Kind P: Establishing Assessment Scales Using a Novel Disciplinary Rationale for Scientific Reasoning. Journal of Research in Science Teaching 2013, 50: 530–560. doi:10.1002/tea.21086 10.1002/tea.21086

Klahr D, Dunbar K: Dual Space Search during Scientific Reasoning. Cognitive Science 1988, 12: 1–48. doi:10.1207/s15516709cog1201_1 10.1207/s15516709cog1201_1

Koeppen K, Hartig J, Klieme E, Leutner D: Current issues in competence modeling and assessment. Journal of Psychology 2008, 216: 61–73. doi:10.1027/0044–3409.216.2.61

Kuhn D: Do students need to be taught how to reason? Educational Research Review 2009, 4: 1–6. doi:10.1016/j.edurev.2008.11.001 10.1016/j.edurev.2008.11.001

Kuo E, Hull M, Gupta A, Elby A: How Students Blend Conceptual and Formal Mathematical Reasoning in Solving Physics Problems. Science Education 2013, 97: 32–57. doi:10.1002/sce.21043 10.1002/sce.21043

Lang M, Olson J: Integrated science teaching as a challenge for teachers to develop new conceptual structures. Research in Science Education 2000, 30: 213–224. doi:10.1007/BF02461629 10.1007/BF02461629

Leutner D, Fleischer J, Wirth J, Greiff S, Funke J: Analytische und dynamische Problemlösekompetenz im Lichte internationaler Schulleistungsstudien [Analytical and dynamic problem-solvng competence in international large-scale studies]. Psychologische Rundschau 2012, 63: 34–42. doi:10.1026/0033–3042/a000108 10.1026/0033-3042/a000108

Lynn R, Meisenberg G: National IQs calculated and validated for 108 nations. Intelligence 2010, 38: 353–360. doi:10.1016/j.intell.2010.04.007 10.1016/j.intell.2010.04.007

MacKinnon D, Krull J, Lockwood C: Equivalence of the mediation, confounding, and suppression effect. Prevention Science 2000, 1: 173–181. doi:10.1023/A:1026595011371 10.1023/A:1026595011371

Mansour N: Consistencies and inconsistencies between science teachers’ beliefs and practices. International Journal of Science Education 2013, 35: 1230–1275. doi:10.1080/09500693.2012.743196 10.1080/09500693.2012.743196

Martin M, Mullis I, Gonzalez E, Chrostowski S: TIMSS 2003 International Science Report . IEA, Chestnut Hill, MA; 2004.

Martin AJ, Liem GAD, Mok MMC, Xu J: Problem solving and immigrant student mathematics and science achievement: Multination findings from the Programme for International Student Assessment (PISA). Journal of Educational Psychology 2012, 104: 1054–1073. doi:10.1037/a0029152 10.1037/a0029152

Mayer R: Problem solving and reasoning. In International Encyclopedia of Education . 3rd edition. Edited by: Peterson P, Baker E, McGraw B. Elsevier, Oxford; 2010:273–278. doi:10.1016/B978–0-08–044894–7.00487–5 10.1016/B978-0-08-044894-7.00487-5

Mayer R, Wittrock MC: Problem solving. In Handbook of Educational Psychology . 2nd edition. Edited by: Alexander PA, Winne PH. Lawrence Erlbaum, New Jersey; 2006:287–303.

Muthén B, Muthén L: Mplus 6 . Muthén & Muthén, Los Angeles, CA; 2010.

Successful K-12 STEM Education . National Academies Press, Washington, DC; 2011.

Nentwig P, Rönnebeck S, Schöps K, Rumann S, Carstensen C: Performance and levels of contextualization in a selection of OECD countries in PISA 2006. Journal of Research in Science Teaching 2009, 8: 897–908. doi:10.1002/tea.20338 10.1002/tea.20338

Neumann K, Kauertz A, Fischer H: Quality of Instruction in Science Education. In Second International Handbook of Science Education (Part One . Edited by: Fraser B, Tobin K, McRobbie C. Springer, Dordrecht; 2012:247–258.

The PISA 2003 Assessment Frameworks . OECD, Paris; 2003.

Problem solving for tomorrow’s world . OECD, Paris; 2004.

PISA 2003 Technical Report . OECD, Paris; 2005.

PISA 2012 Results : Creative Problem Solving – Students’ Skills in Tackling Real-Life Problems (Vol. V) . OECD, Paris; 2014.

Olsen RV: An exploration of cluster structure in scientific literacy in PISA: Evidence for a Nordic dimension? NorDiNa. ᅟ 2005, 1 (1):81–94.

Oser F, Baeriswyl F: Choreographies of Teaching: Bridging Instruction to Learning. In Handbook of Research on Teaching . 4th edition. Edited by: Richardson V. American Educational Research Association, Washington, DC; 2001:1031–1065.

Pellegrino JW, Hilton ML: Education for Life and Work – Developing Transferable Knowledge and Skills in the 21st Century . The National Academies Press, Washington, DC; 2012.

Polya G: How to solve it: a new aspect of mathematical method . Princeton University Press, Princeton, NJ; 1945.

Reich J, Hein S, Krivulskaya S, Hart L, Gumkowski N, Grigorenko E: Associations between household responsibilities and academic competencies in the context of education accessibility in Zambia. Learning and Individual Differences 2013, 27: 250–257. doi:10.1016/j.lindif.2013.02.005 10.1016/j.lindif.2013.02.005

Rindermann H: The g-factor of international cognitive ability comparisons: The homogeneity of results in PISA, TIMSS, PIRLS and IQ-tests across nations. European Journal of Personality 2007, 21: 661–706. doi:10.1002/per.634

Rindermann H: Relevance of education and intelligence at the national level for the economic welfare of people. Intelligence 2008, 36: 127–142. doi:10.1016/j.intell.2007.02.002 10.1016/j.intell.2007.02.002

Risch B: Teaching Chemistry around the World . Waxmann, Münster; 2010.

Ross J, Hogaboam-Gray A: Integrating mathematics, science, and technology: effects on students. International Journal of Science Education 1998, 20: 1119–1135. doi:10.1080/0950069980200908 10.1080/0950069980200908

Snijders TAB, Bosker RJ: Multilevel Analysis: An Introduction to Basic and Advanced Multilevel Modeling . 2nd edition. Sage Publications, London; 2012.

Human Development Report 2005 . UNDP, New York, NY; 2005.

Van Merriënboer J: Perspectives on problem solving and instruction. Computers & Education 2013, 64: 153–160. doi:10.1016/j.compedu.2012.11.025 10.1016/j.compedu.2012.11.025

Wood RE, Beckmann JF, Birney D: Simulations, learning and real world capabilities. Education Training 2009, 51 (5/6):491–510. doi:10.1108/00400910910987273 10.1108/00400910910987273

Wu M: The role of plausible values in large-scale surveys. Studies in Educational Evaluation 2005, 31: 114–128. doi:10.1016/j.stueduc.2005.05.005 10.1016/j.stueduc.2005.05.005

Wu M, Adams R: Modelling Mathematics Problem Solving Item Responses using a Multidimensional IRT Model. Mathematics Education Research Journal 2006, 18: 93–113. doi:10.1007/BF03217438 10.1007/BF03217438

Wüstenberg S, Greiff S, Funke J: Complex problem solving: More than reasoning? Intelligence 2012, 40: 1–14. doi:10.1016/j.intell.2011.11.003 10.1016/j.intell.2011.11.003

Zohar A: Scaling up higher order thinking in science classrooms: the challenge of bridging the gap between theory, policy and practice. Thinking Skills and Creativity 2013, 10: 168–172. doi:10.1016/j.tsc.2013.08.001 10.1016/j.tsc.2013.08.001

Download references

Author information

Authors and affiliations.

Centre for Educational Measurement, University of Oslo (CEMO), Oslo, Norway

Ronny Scherer

Faculty of Educational Sciences, University of Oslo, Oslo, Norway

School of Education, Durham University, Durham, UK

Jens F Beckmann

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Ronny Scherer .

Additional information

Competing interests.

The authors declare that they have no competing interests.

Authors’ contributions

RS carried out the analyses, participated in the development of the rationale, and drafted the manuscript. JFB carried out some additional analyses, participated in the development of the rationale, and drafted the manuscript. Both authors read and approved the final manuscript.

Authors’ original submitted files for images

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Authors’ original file for figure 2, authors’ original file for figure 3, rights and permissions.

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( https://creativecommons.org/licenses/by/4.0 ), which permits use, duplication, adaptation, distribution, and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Cite this article.

Scherer, R., Beckmann, J.F. The acquisition of problem solving competence: evidence from 41 countries that math and science education matters. Large-scale Assess Educ 2 , 10 (2014). https://doi.org/10.1186/s40536-014-0010-7

Download citation

Received : 08 June 2014

Accepted : 25 November 2014

Published : 10 December 2014

DOI : https://doi.org/10.1186/s40536-014-0010-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Capability under-utilisation
• Math-science coherence
• Math education
• Problem solving competence
• Science education

Measuring cognitive load with subjective rating scales during problem solving: differences between immediate and delayed ratings

• Published: 21 August 2014
• Volume 43 , pages 93–114, ( 2015 )

Cite this article

• Annett Schmeck 1 ,
• Maria Opfermann 1 ,
• Tamara van Gog 2 ,
• Fred Paas 2 &
• Detlev Leutner 1  

4937 Accesses

112 Citations

1 Altmetric

Explore all metrics

Subjective cognitive load (CL) rating scales are widely used in educational research. However, there are still some open questions regarding the point of time at which such scales should be applied. Whereas some studies apply rating scales directly after each step or task and use an average of these ratings, others assess CL only once after the whole learning or problem-solving phase. To investigate if these two approaches are comparable indicators of experienced CL, two experiments were conducted, in which 168 and 107 teacher education university students, respectively, worked through a sequence of six problems. CL was assessed by means of subjective ratings of mental effort and perceived task difficulty after each problem and after the whole process. Results showed that the delayed ratings of both effort and difficulty were significantly higher than the average of the six ratings made during problem solving. In addition, the problems we assumed to be of higher complexity seemed to be the best predictors for the delayed ratings. Interestingly, for ratings of affective variables, such as interest and motivation, the delayed rating did not differ from the average of immediate ratings.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Similar content being viewed by others

Comparing Mental Effort, Difficulty, and Confidence Appraisals in Problem-Solving: A Metacognitive Perspective

Emely Hoch, Yael Sidi, … Katharina Scheiter

Looking at Mental Effort Appraisals through a Metacognitive Lens: Are they Biased?

Katharina Scheiter, Rakefet Ackerman & Vincent Hoogerheide

Differentiating Different Types of Cognitive Load: a Comparison of Different Measures

Andreas Korbach, Roland Brünken & Babette Park

Antonenko, P., & Niederhauser, D. (2010). The influence of leads on cognitive load and learning in a hypertext-assisted learning environment. Computers in Human Behavior, 26 , 140–150.

Article   Google Scholar  

Antonenko, P., Paas, F., Grabner, R., & Van Gog, T. (2010). Using electroencephalography (EEG) to measure cognitive load. Educational Psychology Review, 22 , 425–438.

Ayres, P. (2006). Using subjective measures to detect variations of intrinsic load within problems. Learning and Instruction, 16 , 389–400.

Bratfisch, O., Borg, G., & Dornic, S. (1972). Perceived item difficulty in three tests of intellectual performance capacity . Stockholm: Institute of Applied Psychology, Report No. 29.

Brünken, R., Plass, J. L., & Leutner, D. (2003). Direct measurement of cognitive load in multimedia learning. Educational Psychologist, 38 , 53–61.

Brünken, R., Plass, J. L., & Leutner, D. (2004). Assessment of cognitive load in multimedia learning with dual-task methodology: Auditory load and modality effects. Instructional Science, 32 , 115–132.

Chandler, P., & Sweller, J. (1991). Cognitive load theory and the format of instruction. Cognition and Instruction, 8 , 293–332.

Chandler, P., & Sweller, J. (1992). The split-attention effect as a factor in the design of instruction. British Journal of Educational Psychology, 62 , 233–246.

Chandler, P., & Sweller, J. (1996). Cognitive load while learning to use a computer program. Applied Cognitive Psychology, 10 , 1–20.

de Jong, T. (2010). Cognitive load theory, educational research, and instructional design: Some food for thought. Instructional Science, 38 , 105–134.

Kalyuga, S., Chandler, P., & Sweller, J. (1999). Managing split-attention and redundancy in multimedia instruction. Applied Cognitive Psychology, 13 , 351–371.

Kalyuga, S., Chandler, P., Tuovinen, J., & Sweller, J. (2001). When problem solving is superior to studying worked examples. Journal of Educational Psychology, 93 , 579–588.

Kühl, T., Scheiter, K., Gerjets, P., & Edelmann, J. (2011). The influence of text modality on learning with static and dynamic visualizations. Computers in Human Behavior, 27 , 29–35.

Kuusela, H., & Paul, P. (2000). A comparison of concurrent and retrospective verbal protocol analysis. American Journal of Psychology, 113 , 387–404.

Leutner, D., Leopold, C., & Sumfleth, E. (2009). Cognitive load and science text comprehension: Effects of drawing and mentally imagining text content. Computers in Human Behavior, 25 , 284–289.

Logie, R. H. (1995). Visuo-spatial working memory . Hove, UK: Lawrence Erlbaum Associates.

Google Scholar  

Marcus, N., Cooper, M., & Sweller, J. (1996). Understanding instructions. Journal of Educational Psychology, 88 , 49–63.

Opfermann, M. (2008). There’s more to it than instructional design: The role of individual learner characteristics for hypermedia learning . Berlin, Germany: Logos.

Paas, F. (1992). Training strategies for attaining transfer of problem-solving skill in statistics: A cognitive-load approach. Journal of Educational Psychology, 84 , 429–434.

Paas, F., Ayres, P., & Pachman, M. (2008). Assessment of cognitive load in multimedia learning: Theory, methods and applications. In D. Robinson & G. Schraw (Eds.), Recent innovations in educational technology that facilitate student learning (pp. 20–36). Charlotte, NC: Information Age Publishing.

Paas, F., Renkl, A., & Sweller, J. (2003a). Cognitive load theory and instructional design: Recent developments. Educational Psychologist, 38 , 1–4.

Paas, F., Tuovinen, J., Tabbers, H. K., & Van Gerven, P. W. M. (2003b). Cognitive load measurement as a means to advance cognitive load theory. Educational Psychologist, 38 , 63–71.

Paas, F., & van Merriënboer, J. J. G. (1994). Variability of worked examples and transfer of geometrical problem solving skills: A cognitive-load approach. Journal of Educational Psychology, 86 , 122–133.

Paas, F., van Merriënboer, J. J. G., & Adam, J. J. (1994). Measurement of cognitive load in instructional research. Perceptual and Motor Skills, 79 , 419–430.

Park, B., Moreno, R., Seufert, T., & Brünken, R. (2011). Does cognitive load moderate the seductive details effect? A multimedia study. Computers in Human Behavior, 27 , 5–10.

Plass, J. L., Moreno, R., & Brünken, R. (Eds.). (2010). Cognitive load: Theory & application . Cambridge: Cambridge University Press.

Renkl, A., Gruber, H., Weber, S., Lerche, T., & Schweizer, K. (2003). Cognitive Load beim Lernen aus Lösungsbeispielen [Cognitive load of learning from worked-out examples]. Zeitschrift für Pädagogische Psychologie, 17 , 93–101.

Rheinberg, F., Vollmeyer, R., & Burns, B. D. (2001). FAM: Ein Fragebogen zur Erfassung aktueller Motivation in Lern- und Leistungssituationen [QCM: A questionnaire to assess current motivation in learning situations]. Diagnostica, 47 , 57–66.

Schwamborn, A., Thillmann, H., Opfermann, M., & Leutner, D. (2011). Cognitive load and instructionally supported learning with provided and learner-generated visualizations. Computer in Human Behavior, 27 , 89–93.

Seufert, T., & Brünken, R. (2006). Cognitive load and the format of instructional aids for coherence formation. Applied Cognitive Psychology, 20 , 321–331.

Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12 , 257–285.

Sweller, J. (1993). Some cognitive processes and their consequences for the organization and presentation of information. Australian Journal of Psychology, 45 , 1–8.

Sweller, J. (2010). Element interactivity and intrinsic, extraneous and germane cognitive load. Educational Psychology Review, 22 , 123–138.

Sweller, J., Ayres, P., & Kalyuga, S. (2011). Cognitive load theory . New York: Springer.

Book   Google Scholar  

Sweller, J., van Merriënboer, J. J. G., & Paas, F. (1998). Cognitive architecture and instructional design. Educational Psychology Review, 10 , 251–296.

Tabbers, H. K., Martens, R. L., & van Merriënboer, J. J. G. (2004). Multimedia instructions and cognitive load theory: Effects of modality and cueing. British Journal of Educational Psychology, 74 , 71–81.

Taylor, K. L., & Dionne, J.-P. (2000). Accessing problem-solving strategy knowledge: The complementary use of concurrent verbal protocols and retrospective debriefing. Journal of Educational Psychology, 92 , 413–425.

Van Gerven, P. W. M., Paas, F., van Merriënboer, J. J. G., & Schmidt, H. G. (2004). Memory load and the cognitive pupillary response in aging. Psychophysiology, 41 , 167–174.

Van Gog, T., & Jarodzka, H. (2013). Eye tracking as a tool to study and enhance cognitive and metacognitive processes in computer-based learning environments. In R. Azevedo & V. Aleven (Eds.), International handbook of metacognition and learning technologies (pp. 143–156). New York: Springer.

Van Gog, T., Kirschner, F., Kester, L., & Paas, F. (2012). Timing and frequency of mental effort measurement: Evidence in favor of repeated measures. Applied Cognitive Psychology, 26 , 833–839.

Van Gog, T., & Paas, F. (2008). Instructional efficiency: Revisiting the original construct in educational research. Educational Psychologist, 43 , 16–26.

Van Gog, T., Paas, F., & van Merriënboer, J. J. G. (2006). Effects of process-oriented worked examples on troubleshooting transfer performance. Learning and Instruction, 16 , 154–164.

Van Merriënboer, J. J. G., & Sweller, J. (2005). Cognitive load theory and complex learning: Recent developments and future directions. Educational Psychology Review, 17 , 147–177.

Download references

Author information

Authors and affiliations.

Department of Instructional Psychology, Faculty for Educational Sciences, University of Duisburg-Essen, 45117, Essen, Germany

Annett Schmeck, Maria Opfermann & Detlev Leutner

Capaciteitsgroep Psychologie, Faculteit der Sociale Wetenschappen, Erasmus University Rotterdam, Burgemeester Oudlaan 50, T-gebouw, P.O. Box 1738, 3000 DR, Rotterdam, The Netherlands

Tamara van Gog & Fred Paas

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Annett Schmeck .

Weekday problems (according to order of presentation in Experiment 1)

Suppose today is Tuesday. What day of the week is the day after tomorrow?

Suppose today is Thursday. What day of the week is in three days?

Suppose yesterday was Wednesday. What day of the week was four days before the day before yesterday?

What day was yesterday if the day after the day after tomorrow is three days before Sunday?

Suppose five days after the day before yesterday is Friday. What day of the week is tomorrow?

Suppose last Tuesday was the 2nd day. What day of the week is it in 17 days, if the 8th day is in two days?

Rights and permissions

Reprints and permissions

About this article

Schmeck, A., Opfermann, M., van Gog, T. et al. Measuring cognitive load with subjective rating scales during problem solving: differences between immediate and delayed ratings. Instr Sci 43 , 93–114 (2015). https://doi.org/10.1007/s11251-014-9328-3

Download citation

Received : 28 November 2013

Accepted : 21 July 2014

Published : 21 August 2014

Issue Date : January 2015

DOI : https://doi.org/10.1007/s11251-014-9328-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Cognitive load
• Problem solving
• Mental effort
• Task difficulty
• Measurement
• Find a journal
• Publish with us
• Track your research

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

The most recent updated version of this chapter is at: https://opentext.ku.edu/teams/chapter/performance-evaluation/

Introduction

Content in this chapter is available from the Open Stax Exchange.

Learning Outcomes

After reading this chapter, you should be able to answer these questions:

How do organizations effectively use performance appraisals to improve individual job performance, and what are the limitations inherent in the use of various appraisal systems?

What practices are used in the performance appraisal process?

• How do managers give effective feedback to subordinates?

How do organizations choose the best appraisal system for their organization?

How do managers and organizations use incentives and rewards effectively to secure the best possible performance from employees?

EXPLORING MANAGERIAL CAREERS

Two performance appraisal interviews.

“Janet, thanks for coming in. As you know, it’s that time of year again. I’ve been going over this performance appraisal form and have written in my evaluation. I’d like you to look it over and then sign it.”

Janet looked over her ratings, which were nearly all in the “satisfactory” range. Even the category of dependability was marked “satisfactory”; yet, it was Janet who came in on three different occasions to cover for workers in her group who were absent for one reason or another. Janet mentioned this issue to her boss, Ken.

“Well, Janet, you’re right and that’s exactly what I expect of my employees. You know this is your first year here and you can’t expect to reach the top in one jump. But I like your style and if you keep it up, who knows how far you’ll go.”

Twenty-four minutes after the interview began, Janet left, bewildered and disappointed. She had worked hard during her first year; in fact, she had gone the extra mile on a few occasions, and now she was more confused than ever about what was expected of her and what constituted good performance. “Maybe it just doesn’t pay to work hard.”

Two weeks before their scheduled interview, Mary asked Ron to review his goals and accomplishments for the last six months and to note any major changes in his job that had taken place during that period. In the meantime, Mary pulled out the file in which she had periodically recorded both positive and negative specific incidents over the last six months concerning Ron’s performance. She also reviewed the goals they had jointly set at the end of the last review and thought carefully about not only the possible goals for the next six months but longer-term development needs and goals that might be appropriate for Ron.

On the day of the interview, both Mary and Ron came well prepared to review the past six months as well as to think about and plan for the next performance period and beyond. The interview took nearly two hours. After candidly discussing Ron’s past performance and the extent to which both sides felt he had or had not accomplished the goals for that period, they began to focus on what should be accomplished in the future. The discussion caused both sides to make changes in their original evaluations and ideas about targets for the future. When it was over, Ron left more motivated than before and confident that even though he had areas in which he could improve, he had a bright future ahead of him if he continued to be motivated and work hard.

Performance Appraisal Systems

Performance appraisals  are one of the most important and often one of the most mishandled aspects of management. Typically, we think of performance appraisals as involving a boss evaluating a subordinate. However, performance appraisals increasingly involve subordinates appraising bosses through a feedback process known as 360 feedback, 1  customers appraising providers, and peers evaluating coworkers.

Whether appraisals are done by subordinates, peers, customers, or superiors, the process itself is vital to the lifeblood of the organization. Performance appraisal systems provide a means of systematically evaluating employees across various performance dimensions to ensure that organizations are getting what they pay for. They provide valuable feedback to employees and managers, and they assist in identifying promotable people as well as problems. However, such appraisals are meaningless unless they are accompanied by an effective feedback system that ensures that the employee gets the right messages concerning performance.

Reward systems represent a powerful motivational force in organizations, but this is true only when the system is fair and tied to performance. Because a variety of approaches to appraising performance exists, managers should be aware of the advantages and disadvantages of each. In turn, an understanding of reward systems will help managers select the system best suited to the needs and goals of the organization.

Performance appraisal systems serve a variety of functions of central importance to employees. Appraisal techniques practiced today are not without problems, though. Managers should keep abreast of recent developments in compensation and reward systems so they can modify existing systems when more appropriate alternatives become available.

A key management responsibility has always been to oversee and develop subordinates. In fact, it has been said that every manager is a human resource manager. Nowhere is this truer than with regard to evaluating and rewarding subordinates. Managers are consistently involved with employee training and development, monitoring employee performance, providing job-related feedback, and administering rewards.

In this chapter, we examine three interrelated aspects of the performance appraisal and reward process. As  Figure 2  shows, this process moves from evaluating employee performance to providing adequate and constructive feedback to determining discretionary rewards. Where effort and performance are properly evaluated and rewarded, we would expect to see more stable and consistent job performance. On the other hand, where such performance is only evaluated intermittently or where the appraisal and review process is poorly done, we would generally see less consistent performance. We begin our discussion with a look at the nature of appraisals.

We begin by examining three aspects of performance appraisal systems: (1) the uses of performance appraisals, (2) problems found in performance appraisals, and (3) methods for reducing errors in the appraisal system. This overview will provide a foundation for studying specific techniques of performance appraisal. Those interested in more detailed information on performance appraisal systems may wish to consult books on personnel administration or compensation.

Uses of Performance Appraisals

In most work organizations, performance appraisals are used for a variety of reasons. These reasons range from improving employee productivity to developing the employees themselves. This diversity of uses is well documented in a study of why companies use performance appraisals. 2 Traditionally, compensation and performance feedback have been the most prominent reasons organizations use performance appraisals.

Feedback to employees.  Performance appraisals provide feedback to employees about quantity and quality of job performance. Without this information, employees have little knowledge of how well they are doing their jobs and how they might improve their work.

Self-development.  Performance appraisals can also serve as an aid to employee self-development. Individuals learn about their strengths and weaknesses as seen by others and can initiate self-improvement programs (see discussion on behavioral self-management programs).

Reward systems.  In addition, appraisals may form the bases of organizational reward systems—particularly merit-based compensation plans.

Personnel decisions.  Performance appraisals serve personnel-related functions as well. In making personnel decisions, such as those relating to promotions, transfers, and terminations, they can be quite useful. Employers can make choices on the basis of information about individual talents and shortcomings. In addition, appraisal systems help management evaluate the effectiveness of its selection and placement functions. If newly hired employees generally perform poorly, managers should consider whether the right kind of people are being hired in the first place.

Training and development.  Finally, appraisals can help managers identify areas in which employees lack critical skills for either immediate or future performance. In these situations, new or revised training programs can be established to further develop the company’s human resources.

It is apparent that performance appraisal systems serve a variety of functions in organizations. In light of the importance of these functions, it is imperative that the accuracy and fairness of the appraisal be paramount considerations in the evaluation of a system. Many performance appraisal systems exist. It is the manager’s job to select the technique or combination of techniques that best serves the particular needs (and constraints) of the organization. Before considering these various techniques, let us look at some of the more prominent problems and sources of error that are common to several of them.

Problems with Performance Appraisals

A number of problems can be identified that pose a threat to the value of appraisal techniques. Most of these problems deal with the related issues of the validity and reliability of the instruments or techniques themselves.  Validity  is the extent to which an instrument actually measures what it intends to measure, whereas  reliability  is the extent to which the instrument consistently yields the same results each time it is used. Ideally, a good performance appraisal system will exhibit high levels of both validity and reliability. If not, serious questions must be raised concerning the utility (and possibly the legality) of the system.

It is possible to identify several common sources of error in performance appraisal systems. These include: (1) central tendency error, (2) strictness or leniency error, (3) halo effect, (4) recency error, and (5) personal biases.

Central Tendency Error.  It has often been found that supervisors rate most of their employees within a narrow range. Regardless of how people actually perform, the rater fails to distinguish significant differences among group members and lumps everyone together in an “average” category. This is called  central tendency error  and is shown in  Figure 3 . In short, the central tendency error is the failure to recognize either very good or very poor performers.

Strictness or Leniency Error.  A related rating problem exists when a supervisor is overly strict or overly lenient in evaluations (see  Figure 3 ). In college classrooms, we hear of professors who are “tough graders” or, conversely, “easy A’s.” Similar situations exist in the workplace, where some supervisors see most subordinates as not measuring up to their high standards, whereas other supervisors see most subordinates as deserving of a high rating. As with central tendency error,  strictness error  and  leniency error  fail to distinguish adequately between good and bad performers and instead relegate almost everyone to the same or related categories.

Halo Effect.  The  halo effect  exists where a supervisor assigns the same rating to each factor being evaluated for an individual. For example, an employee rated above average on quantity of performance may also be rated above average on quality of performance, interpersonal competence, attendance, and promotion readiness. In other words, the supervisor cannot effectively differentiate between relatively discrete categories and instead gives a global rating.

These types of bias are based on our perceptions of others. The halo effect occurs when managers have an overly positive view of a particular employee. This can impact the objectivity of reviews, with managers consistently giving an employee high ratings and failing to recognize areas for improvement.

Whether positive or negative, we also have a natural tendency to confirm our preconceived beliefs about people in the way we interpret or recall performance, which is known as confirmatory bias.

For example, a manager may have a preconception that her male report is more assertive. This could cause her to recall instances more easily in which her report asserted his position during a meeting. On the other hand, she may perceive her female report to be less assertive, predisposing her to forget when the report suggested an effective strategy or was successful in a tough negotiation.

The halo effect is often a consequence of people having a similarity bias for certain types of people. We naturally tend to favor and trust people who are similar to us. Whether it’s people who also have a penchant for golf or people who remind us of a younger version of ourselves, favoritism that results from a similarity bias can give certain employees an unfair advantage over others. This can impact a team to the point that those employees may receive more coaching, better reviews and, as a result, more opportunities for advancement. 3

Recency Error.  Oftentimes evaluators focus on an employee’s most recent behavior in the evaluation process. This is known as the  recency error . That is, in an annual evaluation, a supervisor may give undue emphasis to performance during the past months—or even weeks—and ignore performance levels prior to this. This practice, if known to employees, leads to a situation where employees may “float” for the initial months of the evaluation period and then overexert themselves in the last few months or weeks prior to evaluation. This practice leads to uneven performance and contributes to the attitude of “playing the game.”

Personal Biases.  Finally, it is not uncommon to find situations in which supervisors allow their own personal biases to influence their appraisals. Such biases include like or dislike for someone, as well as racial and sexual biases. Personal biases can interfere with the fairness and accuracy of an evaluation and are illegal in many situations.

Reducing Errors in Performance Appraisals

A number of suggestions have been advanced recently to minimize the effects of various biases and errors on the performance appraisal process. 4  When errors are reduced, more accurate information is available for personnel decisions and personal development. These methods for reducing error include

• ensuring that each dimension or factor on a performance appraisal form represents a single job activity instead of a group of job activities.
• avoiding terms such as  average , because different evaluators define the term differently.
• ensuring that raters observe subordinates on a regular basis throughout the evaluation period. It is even helpful if the rater takes notes for future reference.
• keeping the number of persons evaluated by one rater to a reasonable number. When one person must evaluate many subordinates, it becomes difficult to discriminate. Rating fatigue increases with the number of ratees.
• ensuring that the dimensions used are clearly stated, meaningful, and relevant to good job performance.
• training raters so they can recognize various sources of error and understand the rationale underlying the evaluation process.

Using mechanisms like these, better employee ratings that can have greater meaning both for the individual employee and the organization will result.

CONCEPT CHECK

• What are performance appraisals, and how are they used in organizations?
• How are performance appraisals used as a reward system, and what problems can they cause?

Techniques of Performance Appraisal

Organizations use numerous methods to evaluate personnel. We will summarize several popular techniques. Although countless variations on these themes can be found, the basic methods presented provide a good summary of the commonly available techniques. Following this review, we will consider the various strengths and weaknesses of each technique. Six techniques are reviewed here: (1) graphic rating scales, (2) critical incident technique, (3) behaviorally anchored rating scales, (4) behavioral observation scales, (5) management by objectives, and (6) assessment centers.

Graphic Rating Scales

Certainly, the most popular method of evaluation used in organizations today is the  graphic rating scale . One study found that 57 percent of the organizations surveyed used rating scales, and another study found the figure to be 65 percent. 5  Although this method appears in many formats, the supervisor or rater is typically presented with a printed or online form that contains both the employee’s name and several evaluation dimensions (quantity of work, quality of work, knowledge of job, attendance). The rater is then asked to rate the employee by assigning a number or rating on each of the dimensions. An example of a graphic rating scale is shown in  Table 1 .

By using this method, if we assume that evaluator biases can be minimized, it is possible to compare employees objectively. It is also possible to examine the relative strengths and weaknesses of a single employee by comparing scores on the various dimensions.

However, one of the most serious drawbacks of this technique is its openness to central tendency, strictness, and leniency errors. It is possible to rate almost everyone in the middle of the scale or, conversely, at one end of the scale. In order to control for this, some companies have assigned required percentage distributions to the various scale points. Supervisors may be allowed to rate only 10 percent of their people outstanding and are required to rate 10 percent unsatisfactory, perhaps assigning 20 percent, 40 percent, and 20 percent to the remaining middle categories. By doing this, a distribution is forced within each department. However, this procedure may penalize a group of truly outstanding performers or reward a group of poor ones.

Critical Incident Technique

With the  critical incident technique  of performance appraisal, supervisors record incidents, or examples, of each subordinate’s behavior that led to either unusual success or unusual failure on some aspect of the job. These incidents are recorded in a daily or weekly log under predesignated categories (planning, decision-making, interpersonal relations, report writing). The final performance rating consists of a series of descriptive paragraphs or notes about various aspects of an employee’s performance (see  Table 2 ).

Job Knowledge —Technical and/or Specialized—possible considerations:

• shows exceptional knowledge in methods, materials, and techniques; applies in a resourceful and practical manner
• stays abreast of development(s) in field and applies to job
• “keeps up” on latest material in her special field
• participates in professional or technical organizations pertinent to her activities

Performance on Human Relations

Ability to Communicate —possible considerations:

• gives logical, clear-cut, understandable instructions on complex problems
• uses clear and direct language in written and oral reporting
• organizes presentations in logical order and in order of importance
• provides supervisor and subordinates with pertinent and adequate information
• tailors communications approach to group or individual
• keeps informed on how subordinates think and feel about things

Results Achieved through Others —possible considerations:

• develops enthusiasm in others that gets the job done
• has respect and confidence of others
• recognizes and credits skills of others
• coordinates well with other involved groups to get the job done

The critical incident method provides useful information for appraisal interviews, and managers and subordinates can discuss specific incidents. Good qualitative information is generated. However, because little quantitative data emerge, it is difficult to use this technique for promotion or salary decisions. The qualitative output here has led some companies to combine the critical incident technique with one of the quantitative techniques, such as the rating scale, to provide different kinds of feedback to the employees.

Behaviorally Anchored Rating Scales

An appraisal system that has received increasing attention in recent years is the  behaviorally anchored rating scale  (BARS). This system requires considerable work prior to evaluation but, if the work is carefully done, can lead to highly accurate ratings with high inter-rater reliability. Specifically, the BARS technique begins by selecting a job that can be described in observable behaviors. Managers and personnel specialists then identify these behaviors as they relate to superior or inferior performance.

An example of this is shown in  Figure 4 , where the BARS technique has been applied to the job of college professor. As shown, as one moves from extremely poor performance to extremely good performance, the performance descriptions, or behavioral anchors, increase. Oftentimes, six to ten scales are used to describe performance on the job. Figure 4  evaluates the professor’s organizational skills. Other scales could relate to the professor’s teaching effectiveness, knowledge of the material, availability to students, and fairness in grading. Once these scales are determined, the evaluator has only to check the category that describes what she observes on the job, and the employee’s rating is simultaneously determined. The BARS technique has several purported advantages. In particular, many of the sources of error discussed earlier (central tendency, leniency, halo) should be significantly reduced because raters are considering verbal descriptions of specific behaviors instead of general categories of behaviors, such as those used in graphic rating scales. In addition, the technique focuses on job-related behaviors and ignores less relevant issues such as the subordinate’s personality, race, or gender. This technique should also lead to employees being less defensive during performance appraisals, because the focus of the discussion would be actual measured behaviors, not the person. Finally, BARS can aid in employee training and development by identifying those domains needing most attention.

On the negative side, as noted above, considerable time and effort in designing the forms are required before the actual rating. Because a separate BARS is required for each distinct job, it is only cost-efficient for common jobs. Finally, because the technique relies on observable behaviors, it may have little applicability for such jobs in such areas as research science (and sometimes management), where much of the work is mental and relevant observable behaviors are difficult to obtain.

Behavioral Observation Scales

The  behavioral observation scale  (BOS) is similar to BARS in that both focus on identifying observable behaviors as they relate to performance. It is, however, less demanding of the evaluator. Typically, the evaluator is asked to rate each behavior on a scale from 1 to 5 to indicate the frequency with which the employee exhibits the behavior. Evaluation of an employee’s performance on a particular dimension is derived by summing the frequency ratings for the behaviors in each dimension.

For example, in  Table 3  we can see an example of a form to evaluate a manager’s ability to overcome resistance to change. The rater simply has to circle the appropriate numbers describing observed behaviors and get a summary rating by adding the results. The BOS technique is easier to construct than the BARS and makes the evaluator’s job somewhat simpler. Even so, this is a relatively new technique that is only now receiving some support in industry.

Management by Objectives

A popular technique for evaluating employees who are involved in jobs that have clear quantitative output is  management by objectives  (MBO). Although the concept of MBO encompasses much more than just the appraisal process (incorporating an organization-wide motivation, performance, and control system), we will focus here on its narrower application to evaluating employee performance. MBO is closely related to the goal-setting theory of motivation.

Under MBO, individual employees work with their supervisor to establish goals and objectives for which they will be responsible during the coming year. These goals are stated in clear language and relate to tasks that are within the domain of the employee. An example of these goals for a sales representative is shown in  Table 4 . Following a specified period of time, the employee’s performance is compared to the preset goals to determine the extent to which the goals have been met or exceeded.

Several advantages of MBO have been observed. These include the ability to do better planning; improved motivation, because of knowledge of results; fairer evaluations, done on the basis of results rather than personality; improved commitment through participation; and improved supervisory skills in such areas as listening, counseling, and evaluating. On the negative side, however, MBO has been criticized because it emphasizes quantitative goals at the expense of qualitative goals and often creates too much paperwork. It is difficult to compare performance levels among employees because most are responsible for different goals. Sometimes the implementation of MBO goals are autocratic and therefore ineffective or even counterproductive. As discussed in the study of motivation, goals must be accepted to be effective. Finally, in order to be successful, MBO implementation must have constant attention and support from top management; MBO does not run itself. In the absence of this support, the technique loses legitimacy and often falls into disrepair.

Assessment Centers

A relatively new method of evaluation is the  assessment center .  Assessment centers are unique among appraisal techniques in that they focus more on evaluating an employee’s long-range potential to an organization than on her performance over the past year. They are also unique in that they are used almost exclusively among managerial personnel.

An assessment center consists of a series of standardized evaluations of behavior based on multiple inputs. Over a two- or three-day period (away from the job), trained observers make judgments on managers’ behavior in response to specially developed exercises. These exercises may consist of in-basket exercises, role-playing, and case analyses, as well as personal interviews and psychological tests. An example of an assessment center program is shown in  Table 5 .

On the basis of these exercises, the trained observers make judgments on employees’ potential for future managerial assignments in the organization. More specifically, information is obtained concerning employees’ interpersonal skills, communication ability, creativity, problem-solving skills, tolerance for stress and ambiguity, and planning ability. This technique has been used successfully by some of the largest corporations in the United States, including AT&T, IBM, and General Electric.

Results from a series of assessment center programs appear promising, and the technique is growing in popularity as a means of identifying future managerial potential. For example, Coca-Cola USA experimented with using assessment centers to select its managerial personnel. After a detailed study, the company found that those selected in this way were only one-third as likely to leave the company or be fired than those selected in the traditional way. Although the assessment center approach added about 6 percent to the cost of hiring, the lower turnover rate led to large overall savings. 6

Some problems with the technique have been noted. In particular, because of the highly stressful environment created in assessment centers, many otherwise good managers may simply not perform to their potential. Moreover, the results of a poor evaluation in an assessment center may be far-reaching; individuals may receive a “loser” image that will follow them for a long time. And, finally, there is some question concerning exactly how valid and reliable assessment centers really are in predicting future managerial success. 7 Despite these problems, assessment centers remain a popular vehicle in some companies for developing and appraising managerial potential.

ETHICS IN PRACTICE

Tesla’s performance review.

At Tesla, the automotive giant, the standards are set extremely high for their employees. In 2017, Tesla conducted its annual performance reviews as it does each year. Due to the review process, the company sees both voluntary and involuntary departures. During the review process, the managers discuss “results that were achieved, as well as how those results were achieved” with their employees.* Tesla also has a performance recognition and compensation program that includes equity rewards as well as promotions in some cases, along with the constructive feedback.

The departure of employees during the review period is not unique to Tesla; however, in 2017 there was a large exodus of approximately 700 employees following their employee reviews. Elon Musk, who recently has stepped down from the role of chairman and has been under scrutiny for his behavior,* saw the media coverage of this news as “ridiculous.”

“You have two boxes of equal ability, and one’s much smaller, the big guy’s going to crush the little guy, obviously,” states Musk. “So, the little guy better have a heck of a lot more skill or he’s going to get clobbered. So that is why our standards are high . . . if they’re not high, we will die.”

Overall, approximately 17 percent of their employees were promoted, almost half in manufacturing. As Tesla continues to grow and develop new vehicles, it is consistently pushing the boundaries and pushing its employees to new limits. Performance reviews are of the highest importance for Tesla’s business to succeed; the company needs the best people with the best skills. It is constantly growing and attempting to “suck the labor pool dry” to fill positions at many of its locations and factories.

• What factors do you feel could have changed in Tesla’s approach to its performance reviews?
• How can a high-pressure environment affect an employee’s performance? What factors should be considered to combat these issues?

Comparison of Appraisal Techniques

It is important to consider which appraisal technique or set of techniques may be most appropriate for a given situation. Although there is no simple answer to this question, we can consider the various strengths and weaknesses of each technique. This is done in  Table 6 . It is important to keep in mind that the appropriateness of a particular appraisal technique is in part a function of the purpose for the appraisal. For example, if the purpose of the appraisal is to identify high potential executives, then assessment centers are more appropriate than rating scales.

As would be expected, the easiest and least expensive techniques are also the least accurate. They are also the least useful for purposes of personnel decisions and employee development. Once again, it appears that managers and organizations get what they pay for. If performance appraisals represent an important aspect of organizational life, clearly the more sophisticated—and more time-consuming—techniques are preferable. If, on the other hand, it is necessary to evaluate employees quickly and with few resources, techniques such as the graphic rating scale may be more appropriate. Managers must make cost-benefit decisions about the price (in time and money) they are willing to pay for a quality performance appraisal system.

• What are the techniques and scales used in performance appraisals?
• What are MBOs, and how do they relate to performance appraisals?
• What are assessment centers?
• What types of feedback do performance appraisals provide to all organization members?

Reward Systems in Organizations

After a company has designed and implemented a systematic performance appraisal system and provided adequate feedback to employees, the next step is to consider how to tie available corporate rewards to the outcomes of the appraisal. Behavioral research consistently demonstrates that performance levels are highest when rewards are contingent upon performance. Thus, in this section, we will examine five aspects of reward systems in organizations: (1) functions served by reward systems, (2) bases for reward distribution, (3) intrinsic versus extrinsic rewards, (4) the relationship between money and motivation and, finally, (5) pay secrecy.

Functions of Reward Systems

Reward systems in organizations are used for a variety of reasons. It is generally agreed that reward systems influence the following:

• Job effort and performance.   Following expectancy theory, employees’ effort and performance would be expected to increase when they felt that rewards were contingent upon good performance. Hence, reward systems serve a very basic motivational function.
• Attendance and retention.   Reward systems have also been shown to influence an employee’s decision to come to work or to remain with the organization. This was discussed in the previous chapter.
• Employee commitment to the organization.   It has been found that reward systems in no small way influence employee commitment to the organization, primarily through the exchange process. 9 That is, employees develop ties with organizations when they perceive that the organization is interested in their welfare and willing to protect their interests. This exchange process is shown in Figure 6 To the extent that employee needs and goals are met by the company, we would expect commitment to increase.
• Job satisfaction . Job satisfaction has also been shown to be related to rewards, as discussed in the previous chapter. Edward E. Lawler, a well-known researcher on employee compensation, has identified four conclusions concerning the relationship between rewards and satisfaction: (1) satisfaction with a reward is a function of both how much is received and how much the individual feels should have been received; (2) satisfaction is influenced by comparisons with what happens to others, especially one’s coworkers; (3) people differ with respect to the rewards they value; and (4) some rewards are satisfying because they lead to other rewards. 10
• Occupational and organizational choice.   Finally, the selection of an occupation by an individual, as well as the decision to join a particular organization within that occupation, are influenced by the rewards that are thought to be available in the occupation or organization. To prove this, simply look at the classified section of your local newspaper and notice how many jobs highlight beginning salaries.

Reward systems in organizations have far-reaching consequences for both individual satisfaction and organizational effectiveness. Unfortunately, cases can easily be cited where reward systems have been distorted to punish good performance or inhibit creativity. Consider, for example, the Greyhound Bus Company driver who was suspended for 10 days without pay for breaking a company rule against using a CB radio on his bus. The bus driver had used the radio to alert police that his bus, with 32 passengers on board, was being hijacked by an armed man. The police arrested the hijacker, and the bus driver was suspended for breaking company rules. 11 Such incidents hardly encourage employees to focus their efforts on responsible performance.

Bases for Reward Distribution

A common reality in many contemporary work organizations is the inequity that exists in the distribution of available rewards. One often sees little correlation between those who perform well and those who receive the greatest rewards. At the extreme, it is hard to understand how a company could pay its president $10 to $20 million per year (as many large corporations do) while it pays its secretaries and clerks less than $15,000. Each works approximately 40 hours per week, and both are important for organizational performance. Is it really possible that the president is 1,000 times more important than the secretary, as the salary differential suggests?

How do organizations decide on the distribution of available rewards? At least four mechanisms can be identified. In more cases than we choose to admit, rewards go to those with the greatest  power  (either market power or personal power). In many of the corporations whose presidents earn eight-figure incomes, we find that these same people are either major shareholders in the company or have certain abilities, connections, or status that the company wants. Indeed, a threat of resignation from an important or high-performing executive often leads to increased rewards.

A second possible basis for reward distribution is  equality . Here, all individuals within one job classification would receive the same, or at least similar, rewards. The most common example here can be found among unionized workers, where pay rates are established and standardized with little or no reference to actual performance level. Instead of ability or performance, these systems usually recognize seniority as the key factor in pay raises or promotions.

The basis for the social welfare reward system in this country is need. In large part, the greater the need, the greater the level of support. It is not uncommon to see situations in business firms where need is taken into account in layoff situations—where an employee is not laid off because she is the sole support of a family.

A fourth mechanism used by organizations in allocating rewards is  distributive justice . Under this approach, employees receive (at least a portion of) their rewards as a function of their level of contribution to the organization. The greater the contribution (such as performance), the greater the reward. This mechanism is most prominent in merit-based incentive programs, where pay and bonuses are determined by performance levels.

Extrinsic and Intrinsic Rewards

The variety of rewards that employees can receive in exchange for their contributions of time and effort can be classified as either  extrinsic  or  intrinsic  rewards.  Extrinsic rewards  are external to the work itself. They are administered externally—that is, by someone else (usually management). Examples of extrinsic rewards include wages and salary, fringe benefits, promotions, and recognition and praise from others.

On the other hand,  intrinsic rewards  represent those rewards that are related directly to performing the job. In this sense, they are often described as “self-administered” rewards, because engaging in the task itself leads to their receipt. Examples of intrinsic rewards include feelings of task accomplishment, autonomy, and personal growth and development that come from the job.

In the literature on employee motivation, there is considerable controversy concerning the possible interrelationship of these two kinds of reward. It has been argued (with some research support) that extrinsic rewards tend to drive out the positive effects of some intrinsic rewards and can lead to unethical behavior. 12  Consider, for example, the child next door who begs you to let her help you wash your car. For a young child, this task can carry considerable excitement (and intrinsic motivation). Now, consider what happens on a Saturday afternoon when you need your car washed but the child has other options. What do you do? You offer to pay her this time to help wash your car. What do you think will happen the next time you ask the neighbor to help you wash the car for free? In other words, when extrinsic rewards such as pay are tied closely to performance (called performance-reward contingency),  intrinsic motivation —the desire to do a task because you enjoy it—can decrease.

Also, it is important to keep in mind that because extrinsic rewards are administered by sources external to the individual, their effectiveness rests on accurate and fair monitoring, evaluating, and administration. Implementation can be expensive, and the timing of performance and rewards may not always be close. For example, you may perform well on a task, but unless there is a way for that to be noticed, evaluated, recorded, and rewarded within a reasonable time frame, an extrinsic reward may not have a significant impact. Intrinsic rewards are a function of self-monitoring, evaluation, and administration; consequently, these rewards often are less costly and more effectively administered. For example, even if no one else notices or rewards you for superior performance on a task, you can still reward yourself with a mental pat on the back for a job well done or a sense of satisfaction for overcoming a challenge. The implications of this finding will become apparent when exploring efforts to enrich employees’ jobs.

Money and Motivation: A Closer Look

A recurring debate among managers focuses on the issue of whether money is a primary motivator. Some argue that most behavior in organizational settings is motivated by money (or at least monetary factors), whereas others argue that money is only one of many factors that motivate performance. Whichever group is correct, we must recognize that money can have important motivational consequences for many people in many situations. In fact, money serves several important functions in work settings. 13  These include serving as (1) a goal or incentive, (2) a source of satisfaction, (3) an instrument for gaining other desired outcomes, (4) a standard of comparison for determining relative standing or worth, and (5) a conditional reinforcer where its receipt is contingent upon a certain level of performance. Even so, experience tells us that the effectiveness of pay as a motivator varies considerably. Sometimes there seems to be an almost direct relationship between pay and effort, whereas at other times no such relationship is found. Why? Lawler suggests that certain conditions must be present in order for pay to act as a strong motivator: 14

• Trust levels between managers and subordinates must be high.
• Individual performance must be able to be accurately measured.
• Pay rewards to high performers must be substantially higher than those to poor performers.
• Few, if any, negative consequences for good performance must be perceived.

Under these conditions, a climate or culture is created in which employees have reason to believe that significant performance-reward contingencies truly exist. Given this perception (and assuming the reward is valued), we would expect performance to be increased. 15

Pay Secrecy

Secrecy about pay rates seems to be a widely accepted practice in work organizations, particularly among managerial personnel. It is argued that salary is a personal matter and we should not invade another’s privacy. Available evidence, however, suggests that pay secrecy may have several negative side effects. To begin, it has been consistently found that in the absence of actual knowledge, people have a tendency to  over estimate the pay of coworkers and those above them in the hierarchy. As a result, much of the motivational potential of a differential reward system is lost. 16  Even if an employee receives a relatively sizable salary increase, she may still perceive an inequity compared to what others are receiving. This problem is highlighted in the results of a study by Lawler. In considering the effects of pay secrecy on motivation, Lawler noted:

Almost regardless of how well the individual manager was performing, he felt he was getting less than the average raise. This problem was particularly severe among high performers, since they believed that they were doing well yet received minimal reward. They did not believe that pay was in fact based upon merit. This was ironic, since their pay did reflect performance. . . . Thus, even though pay was tied to performance, these managers were not motivated because they could not see the connection. 17

Pay secrecy also affects motivation via feedback. Several studies have shown the value of feedback in motivating performance (see previous discussion). The problem is that for managers, money represents one of the most meaningful forms of feedback. Pay secrecy eliminates the feedback.

When salary information is open (or at least when the range of percentage increases within a job classification are made known to the people in that group), employees are generally provided with more recognition for satisfactory performance and are often more motivated to perform on subsequent tasks. It is easier to establish feelings of pay equity and trust in the salary administration system. On the other hand, publicizing pay rates and pay raises can cause jealousy among employees and create pressures on managers to reduce perceived inequities in the system. There is no correct position concerning whether pay rates should be secret or open. The point is that managers should not assume a priori that pay secrecy—or pay openness—is a good thing. Instead, careful consideration should be given to the possible consequences of either approach in view of the particular situation in the organization at the time.

• What is the best appraisal system for organizations to adopt?
• How are rewards tied to performance appraisals?

Individual and Group Incentive Plans

We now turn to an examination of various employee incentive programs used by organizations. First, we consider the relative merits of individuals versus group incentive programs. Next, we focus on several relatively new approaches to motivation and compensation. Finally, we suggest several guidelines for effective incentive systems.

Individual versus Group Incentives

Companies usually have choices among various compensation plans and must make decisions about which is most effective for its situation. Incentive systems in organizations are usually divided into two categories on the basis of whether the unit of analysis—and the recipient of the reward—is the individual or a group. Among individual incentive plans, several approaches can be identified, including merit-based compensation (commonly known as merit compensation), piece-rate incentive programs (where people are paid according to the quantity of output), bonus systems of various sorts, and commissions. In each case, rewards are tied fairly directly to the performance level of the individual.

Although individual incentive systems often lead to improved performance, some reservations have been noted. In particular, these programs may at times lead to employees competing with one another, with undesirable results. For instance, department store salespeople on commission may fight over customers, thereby chasing the customers away. After all, customers don’t care who they deal with, only that the service is good. Second, these plans typically are resisted by unions, which prefer compensation to be based on seniority or job classification. Third, where quality control systems are lax, individual incentives such as piece rates may lead employees to maximize units of output while sacrificing quality. And, finally, in order for these programs to be successful, an atmosphere of trust and cooperation is necessary.

In order to overcome some of these shortcomings, many companies have turned to group or organizational incentive plans. Group incentive programs base at least some of an employee’s rewards on group or organization performance. Hence, employees are encouraged to cooperate with one another and with the corporation so that all employees can benefit. Programs such as profit-sharing or gain-sharing plans (discussed below) are designed to tie the employees’ future rewards and prosperity to that of the company and reduce the age-old antagonism between the two. The results are often dramatic.

Creative Pay Practices

Recently, we have seen several innovations in the way corporations approach reward systems. These efforts are designed to facilitate the integration of employee and company interests in a way that maximizes both productivity and quality of working life. Five such creative pay practices should be noted: (1) gain-sharing plans, (2) skills-based incentives, (3) lump-sum pay increases, (4) participative pay decisions, and (5) flexible benefits programs. These approaches, along with their major advantages and drawbacks, are summarized in  Table 7 .

Gain-Sharing Plans.  Giving executives and senior managers bonuses to reflect their contributions to organizational effectiveness is commonplace. In fact, in some companies executive bonuses are often larger than salaries. Recently, companies have increasingly applied this same principle to all employees in the form of  gain-sharing  (profit-sharing) plans. Here, employees are given a chance to share in corporate productivity gains through increased earnings. The greater the productivity gains, the greater the earnings. Several variations on this theme can be found, including the Scanlon Plan, IMPROSHARE, the Ruker Plan, and the Lincoln Electric Plan. Regardless of the title, the basic plan is similar.

For example, under the Scanlon Plan (probably the oldest such program), three operating guidelines are used: (1) each department or division is treated as a business unit for purposes of performance measurement, (2) specific cost measures associated with the production process are identified and agreed to by all parties, and (3) bonuses are paid to all employees according to a predetermined formula tying the amount of the bonus to the actual cost savings realized during the time period. Under such a plan, it is clearly in the employees’ best interest to contribute to cost savings, thereby increasing their own incomes.

EXPANDING AROUND THE GLOBE

Providing feedback in different countries.

Global workplaces are increasing within the world businesses, and it has become a trend to have managers from one country, most likely the country in which the headquarters arise, manage employees abroad. An important consideration when managing globally is how cultural differences can have a profound effect on performance evaluations, negotiations, and criticisms.

For example, oftentimes in the United States, a method of critical feedback in the “hamburger method” (Step 1: Identify tasks. As a group, identify technical steps that would be involved in implementing. Step 2: Identify options for tasks. Split the team into several small groups. Step 3: Combine results.) is acceptable, while other countries give their feedback with just the meal alone. This strategy in the Netherlands and Germany can be off-putting to other cultures, and when you read into another culture’s technique with your own lens of reference, it can feel wrong.

Managing globally means that you need to do your research on which approach for feedback is best received for the employee’s cultural differences. For example, being direct is key when communicating with a Dutch person. In contrast, in England or the United States, criticism is not delivered directly, but with positive pieces wrapped around the negative. In Asian countries, feedback is often avoided or the message is blurred in order to “save face.” With all of these complications and considerations, it is ever more important to acutely understand the culture, the cultural understandings of employees who are direct reports, and also the lens through which feedback is being viewed as well.

• How can a new manager that is working with international employees ensure she is providing reviews in an appropriate manner?
• What methods can a manager employ in her preparation for the review to be successful when providing feedback to employees of different cultures?

Skills-Based Incentives.  Typical compensation programs are tied to job evaluations. In these, jobs are analyzed to assess their characteristics, and then salary levels are assigned to each job on the basis of factors such as job difficulty and labor market scarcity. In other words, pay levels are set on the basis of the job, not the individual. This approach fails to encourage employees to improve their skills on the job, because there is no reward for the improvement. This thinking also keeps all employees in their places and minimizes the possibility of inter-job transfers.

Under the  skills-based incentive  program ,  employees are paid according to their skills level (that is, the  number  of jobs they can perform), regardless of the actual tasks they are allowed to perform. This approach has proved successful in organizations such as Procter & Gamble and General Foods. Employees are encouraged to learn additional skills and are appropriately rewarded. The organization is provided with a more highly trained and more flexible workforce. However, training and compensation costs are necessarily increased, so the program is appropriate only in some situations. The technique is most often seen as part of a larger quality-of-working-life program, where it is associated with job redesign efforts.

Lump-Sum Pay Increases.  Another technique that has received some attention is to allow employees to decide how (that is, in what amounts) they wish to receive their pay raises for the coming year. Under the traditional program, pay raises are paid in equal amounts in each paycheck over the year. Under the alternate plan, employees can elect to receive equal amounts during the year, or they can choose to take the entire raise in one  lump-sum pay increase . This plan allows employees greater discretion over their own financial matters. If an employee wants to use the entire pay raise for a vacation, it can be paid in a lump sum in June. Then, if the employee quits before the end of the year, the unearned part of the pay raise is subtracted from the final paycheck. This plan increases the visibility of the reward to the employee. The employee receives, for example, a $600 pay raise (a rather sizable amount) instead of twelve $50 monthly pay raises. As with the flexible rewards system discussed below, however, the administration costs of the lump-sum plan are greater than those of the traditional method.

Participative Pay Decisions.  In addition, of concern to many managers is the extent to which employees should be involved in decisions over pay raises. This is the issue of  participative pay decisions . Recently, several organizations have been experimenting with involving employees in pay raise decisions, and the results seem to be quite positive. By allowing employees to participate either in the design of the reward system or in actual pay raise decisions (perhaps through a committee), it is argued that decisions of higher quality are made on the basis of greater information. Also, employees then have greater reason to place confidence in the fairness of the decisions. On the negative side, this approach requires considerably more time for both the manager and the participating subordinates. Costs must be weighed against benefits to determine which approach is most suitable for the particular organization and its goals.

Flexible Benefits Systems . A typical fringe benefit package provides the same benefits—and the same number of benefits—to all employees. As a result, individual differences or preferences are largely ignored. Studies by Lawler indicate variations in benefit preferences. 18  For instance, young unmarried men prefer more vacation time, whereas young married men prefer to give up vacation time for higher pay. Older employees want more retirement benefits, whereas younger employees prefer greater income. Through a  flexible benefits program  (also called a “cafeteria benefits program”), employees are allowed some discretion in the determination of their own packages and can make trade-offs, within certain limits. Organizations such as PepsiCo, TRW, and the Educational Testing Service already use such programs. Although certain problems of administration exist with the programs, efforts in this direction can lead to increased need satisfaction among employees.

Which approaches are most effective in motivating employees? This is obviously a difficult question to answer.  One such study asked major employers which of a variety of approaches had been used with a high success level. The results are shown in  Table 8 . Skills-based compensation, earned time off, and  gain sharing  all received high marks from personnel executives, although other programs are also widely supported. It would appear from these results that many approaches can be useful; the choice of which one to use would depend upon the circumstances and goals of a particular organization.

Guidelines for Effective Incentive Programs

Whatever incentive plan is selected, care must be taken to ensure that the plan is appropriate for the particular organization and workforce. In fact, a simple test of the effectiveness of an incentive plan would be as follows: 19

• Does the plan capture attention?  Do employees discuss the plan and take pride in their early successes?
• Do employees understand the plan?  Can employees explain how the plan works, and do they understand what they must do to earn the incentive?
• Does the plan improve communication?  As a result of the plan, do employees understand more about corporate mission, goals, and objectives?
• Does the plan pay out when it should?  Are incentives being paid for desired results, and are they withheld for undesirable results?
• Is the company performing better as a result of the plan?  Are profits or market share up or down? Have any gains resulted in part from the incentive plan?

If a new (or existing) pay plan can meet these tests, it is probably fairly effective in motivating employee performance and should be retained by the organization. If not, perhaps some other approach should be tried. On the basis of such a test, several specific guidelines can be identified to increase the effectiveness of the programs. These include the following: 20

• Any reward system or incentive plan should be as  closely tied to actual job performance  as possible. This point was discussed earlier in this chapter.
• If possible, incentive programs should  allow for individual differences . They should recognize that different people want different outcomes from a job. Flexible benefits programs such as the ones discussed here make an effort to accomplish this.
• Incentive programs should  reflect the type of work that is done  and the structure of the organization. This simply means that the program should be tailored to the particular needs, goals, and structures of a given organization. Individual incentive programs, for example, would probably be less successful among unionized personnel than would group programs such as the Scanlon plan. This point has been clearly demonstrated in research by Lawler, which points out that organizations with traditional management and those with more participative management might approach reward systems quite differently in order to be effective. 21  As shown in  Table 9 , both types of company can be effective as long as their reward systems are congruent with their overall approach to management.
• The incentive program should  be consistent with the culture  and constraints of the organization. Where trust levels are low, for example, it may take considerable effort to get any program to work. In an industry already characterized by high levels of efficiency, basing an incentive system on increasing efficiency even further may have little effect, because employees may see the task as nearly impossible.
• Finally, incentive programs should  be carefully monitored over time  to ensure that they are being fairly administered and that they accurately reflect current technological and organizational conditions. For instance, it may be appropriate to offer sales clerks in a department store an incentive to sell outdated merchandise because current fashion items sell themselves. Responsibility falls on managers not to select the incentive program that is in vogue or used “next door,” but rather to consider the unique situations and needs of their own organizations. Then, with this understanding, a program can be developed and implemented that will facilitate goal-oriented performance.
• What are the differences between individual and group incentives?
• What is the variety of reward incentives available to organizations?

If performance is to be changed or improved, it must be rewarded. To be rewarded, it must be measured. However, great care must be taken to (1) measure important behaviors and outcomes (individual, group, or organizational) and not just those that are easy to measure, (2) measure them with the appropriate technique(s), and (3) tie appropriate rewards to the desired behaviors and outcomes.

Organizations use performance appraisals for several reasons: (1) to provide feedback to employees, (2) to allow for employee self-development, (3) to allocate rewards, (4) to gather information for personnel decisions, and (5) to guide them in developing training and development efforts.

Performance appraisals are subject to several problems, including central tendency error, strictness or leniency error, halo effect, recency error, and personal biases.

Among the most common appraisal systems are graphic rating scales, critical incident technique, behaviorally anchored rating scales, behavioral observation scales, management by objectives, and assessment centers. Assessment centers represent a special case of evaluations in that they focus on assessing an employee’s long-term potential to an organization.

Rewards serve several functions, including (1) stimulating job effort and performance, (2) reducing absenteeism and turnover, (3) enhancing employee commitment, (4) facilitating job satisfaction, and (5) facilitating occupational and organizational choice.

Rewards may be distributed on the basis of power, equality, need, or distributive justice. Distributive justice rests on the principle of allocating rewards in proportion to employee contribution. Intrinsic rewards represent those outcomes that are administered by the employee (e.g., a sense of task accomplishment), whereas extrinsic rewards are administered by others (e.g., wages).

Gain-sharing incentive plans base some of the employees’ pay on corporate profits or productivity. As a result, employees are generally more interested in facilitating corporate performance.  Skills-based incentives  reward employees on the basis of the skills they possess, not the skills they are allowed to use at work. As a result, employees are encouraged to continually upgrade their skill levels.

A lump-sum salary increase simply provides employees with their pay raises at one time (possibly shortly before summer vacation or a major holiday).

Participative pay decisions allow employees some input in determining their pay raises.

Flexible benefits allow employees to choose the fringe benefits that best suit their needs.

A good reward system (1) is closely tied to performance, (2) allows for individual differences, (3) reflects the type of work that is being done, (4) is consistent with the corporate culture, and (5) is carefully monitored over time.

Chapter Review Questions

• Identify the various functions of performance appraisals. How are appraisals used in most work organizations?
• What are some problems associated with performance appraisals?
• Define  validity  and  reliability . Why are these two concepts important from a managerial standpoint?
• How can errors in appraisals be reduced?
• Critically evaluate the advantages and disadvantages of the various techniques of performance appraisal.
• Discuss the role of feedback in employee performance.
• What is the difference between intrinsic and extrinsic rewards?
• Identify the major bases of reward distribution.
• How does money influence employee motivation?
• Discuss the relative merits of individual and group incentive programs.
• Describe the benefits and drawbacks of several of the new approaches to reward systems. Which ones do you feel would be most effective in work organizations?

Management Skills Application Exercises

How would you rate your supervisor.

Instructions:  Think of your current supervisor or one for any job you have held, and evaluate her on the following dimensions. Give a “1” for very poor, a “3” for average, a “5” for outstanding, etc.

Think of a current or previous job, and evaluate the source and quality of the feedback you received from your supervisor. When you are through, refer to  Appendix B  for scoring procedures.

How Much Feedback Are You Getting from Your Job?

Instructions:  Think of a current or previous job. With this in mind, answer the following questions as accurately as possible.

My boss lets me know when I make a mistake.

My coworkers help me improve on the job.

I receive formal evaluations from the company on my job.

My boss always tells me when I do a good job.

This company really appreciates good performance.

When I do something especially well, I receive a “thanks” from my boss.

My coworkers are very appreciative when I do a good job.

My coworkers compliment me on the quality of my work.

My coworkers are very supportive of my efforts.

I know when I have done a good job.

My job provides me with solid feedback on my performance.

I can see the results when I learn to do something better.

C. Solbach. “Feedback through cultural looking glass.” Krauthammer, September 16, 2015, https://www.krauthammer.com/en/publications/personal-development/2017/04/12/12/07/feedback-through-cultural-looking-glass;

M. Abadi. “The exact same sentence from your boss can mean ‘yes,’ ‘no,’ or ‘maybe’ depending on the country where you work.” Business Insider, December 7, 2017, https://www.businessinsider.com/direct-feedback-work-depends-on-culture-2017-12; J. Windust. “An International Approach to 360-Degree Feedback.”

Cognology, July 26, 2016, https://www.cognology.com.au/international-approach-360-degree-feedback/; “Giving Employee Feedback To A Culturally Diverse Workforce.” Impraise Blog, accessed January 26, 2019, https://blog.impraise.com/360-feedback/how-to-handle-cultural-differences-between-branchescountries-in-feedback-behavior-performance-review.

K. Korosec. “Tesla Fires Hundreds of Workers After Their Annual Performance Review.” Fortune , October 14, 2017,http://fortune.com/2017/10/13/tesla-fires-employees/; D. Muoio. “Tesla fired 700 employees after performance reviews in the third quarter.”  Business Insider , November 1, 2017, https://www.businessinsider.com/tesla-fired-700-employees-performance-reviews-2017-11;

J. Wattles. “Elon Musk agrees to pay $20 million and quit as Tesla chairman in deal with SEC.”  Money , September 30, 2018, https://money.cnn.com/2018/09/29/technology/business/elon-musk-tesla-sec-settlement/index.html.

Problem Solving in Teams and Groups (updated at: https://opentext.ku.edu/teams/) Copyright © by Cameron W. Piercy, Ph.D. is licensed under a Creative Commons Attribution 4.0 International License , except where otherwise noted.

Share This Book

Why Schools Need to Change Yes, We Can Define, Teach, and Assess Critical Thinking Skills

Jeff Heyck-Williams (He, His, Him) Director of the Two Rivers Learning Institute in Washington, DC

Today’s learners face an uncertain present and a rapidly changing future that demand far different skills and knowledge than were needed in the 20th century. We also know so much more about enabling deep, powerful learning than we ever did before. Our collective future depends on how well young people prepare for the challenges and opportunities of 21st-century life.

Critical thinking is a thing. We can define it; we can teach it; and we can assess it.

While the idea of teaching critical thinking has been bandied around in education circles since at least the time of John Dewey, it has taken greater prominence in the education debates with the advent of the term “21st century skills” and discussions of deeper learning. There is increasing agreement among education reformers that critical thinking is an essential ingredient for long-term success for all of our students.

However, there are still those in the education establishment and in the media who argue that critical thinking isn’t really a thing, or that these skills aren’t well defined and, even if they could be defined, they can’t be taught or assessed.

To those naysayers, I have to disagree. Critical thinking is a thing. We can define it; we can teach it; and we can assess it. In fact, as part of a multi-year Assessment for Learning Project , Two Rivers Public Charter School in Washington, D.C., has done just that.

Before I dive into what we have done, I want to acknowledge that some of the criticism has merit.

First, there are those that argue that critical thinking can only exist when students have a vast fund of knowledge. Meaning that a student cannot think critically if they don’t have something substantive about which to think. I agree. Students do need a robust foundation of core content knowledge to effectively think critically. Schools still have a responsibility for building students’ content knowledge.

However, I would argue that students don’t need to wait to think critically until after they have mastered some arbitrary amount of knowledge. They can start building critical thinking skills when they walk in the door. All students come to school with experience and knowledge which they can immediately think critically about. In fact, some of the thinking that they learn to do helps augment and solidify the discipline-specific academic knowledge that they are learning.

The second criticism is that critical thinking skills are always highly contextual. In this argument, the critics make the point that the types of thinking that students do in history is categorically different from the types of thinking students do in science or math. Thus, the idea of teaching broadly defined, content-neutral critical thinking skills is impossible. I agree that there are domain-specific thinking skills that students should learn in each discipline. However, I also believe that there are several generalizable skills that elementary school students can learn that have broad applicability to their academic and social lives. That is what we have done at Two Rivers.

Defining Critical Thinking Skills

We began this work by first defining what we mean by critical thinking. After a review of the literature and looking at the practice at other schools, we identified five constructs that encompass a set of broadly applicable skills: schema development and activation; effective reasoning; creativity and innovation; problem solving; and decision making.

We then created rubrics to provide a concrete vision of what each of these constructs look like in practice. Working with the Stanford Center for Assessment, Learning and Equity (SCALE) , we refined these rubrics to capture clear and discrete skills.

For example, we defined effective reasoning as the skill of creating an evidence-based claim: students need to construct a claim, identify relevant support, link their support to their claim, and identify possible questions or counter claims. Rubrics provide an explicit vision of the skill of effective reasoning for students and teachers. By breaking the rubrics down for different grade bands, we have been able not only to describe what reasoning is but also to delineate how the skills develop in students from preschool through 8th grade.

Before moving on, I want to freely acknowledge that in narrowly defining reasoning as the construction of evidence-based claims we have disregarded some elements of reasoning that students can and should learn. For example, the difference between constructing claims through deductive versus inductive means is not highlighted in our definition. However, by privileging a definition that has broad applicability across disciplines, we are able to gain traction in developing the roots of critical thinking. In this case, to formulate well-supported claims or arguments.

Teaching Critical Thinking Skills

The definitions of critical thinking constructs were only useful to us in as much as they translated into practical skills that teachers could teach and students could learn and use. Consequently, we have found that to teach a set of cognitive skills, we needed thinking routines that defined the regular application of these critical thinking and problem-solving skills across domains. Building on Harvard’s Project Zero Visible Thinking work, we have named routines aligned with each of our constructs.

For example, with the construct of effective reasoning, we aligned the Claim-Support-Question thinking routine to our rubric. Teachers then were able to teach students that whenever they were making an argument, the norm in the class was to use the routine in constructing their claim and support. The flexibility of the routine has allowed us to apply it from preschool through 8th grade and across disciplines from science to economics and from math to literacy.

Kathryn Mancino, a 5th grade teacher at Two Rivers, has deliberately taught three of our thinking routines to students using the anchor charts above. Her charts name the components of each routine and has a place for students to record when they’ve used it and what they have figured out about the routine. By using this structure with a chart that can be added to throughout the year, students see the routines as broadly applicable across disciplines and are able to refine their application over time.

Assessing Critical Thinking Skills

By defining specific constructs of critical thinking and building thinking routines that support their implementation in classrooms, we have operated under the assumption that students are developing skills that they will be able to transfer to other settings. However, we recognized both the importance and the challenge of gathering reliable data to confirm this.

With this in mind, we have developed a series of short performance tasks around novel discipline-neutral contexts in which students can apply the constructs of thinking. Through these tasks, we have been able to provide an opportunity for students to demonstrate their ability to transfer the types of thinking beyond the original classroom setting. Once again, we have worked with SCALE to define tasks where students easily access the content but where the cognitive lift requires them to demonstrate their thinking abilities.

These assessments demonstrate that it is possible to capture meaningful data on students’ critical thinking abilities. They are not intended to be high stakes accountability measures. Instead, they are designed to give students, teachers, and school leaders discrete formative data on hard to measure skills.

While it is clearly difficult, and we have not solved all of the challenges to scaling assessments of critical thinking, we can define, teach, and assess these skills . In fact, knowing how important they are for the economy of the future and our democracy, it is essential that we do.

Jeff Heyck-Williams (He, His, Him)

Director of the two rivers learning institute.

Jeff Heyck-Williams is the director of the Two Rivers Learning Institute and a founder of Two Rivers Public Charter School. He has led work around creating school-wide cultures of mathematics, developing assessments of critical thinking and problem-solving, and supporting project-based learning.

Read More About Why Schools Need to Change

Bring Your Vision for Student Success to Life with NGLC and Bravely

March 13, 2024

For Ethical AI, Listen to Teachers

Jason Wilmot

October 23, 2023

Turning School Libraries into Discipline Centers Is Not the Answer to Disruptive Classroom Behavior

Stephanie McGary

October 4, 2023

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• HHS Author Manuscripts

Efficient Assessment of Social Problem-Solving Abilities in Medical and Rehabilitation Settings: A Rasch Analysis of the Social Problem-Solving Inventory-Revised

Laura e. dreer.

University of Alabama at Birmingham

Patricia Rivera

Marsha snow, timothy r. elliott.

Texas A & M University

Doreen Miller

Southern University

Todd D. Little

University of Kansas

The Social Problem Solving Inventory-Revised Scale (SPSI-R) has been shown to be a reliable and valid self-report measure of social problem-solving abilities. In busy medical and rehabilitation settings, a brief and efficient screening version with psychometric properties similar to the SPSI-R would have numerous benefits including decreased patient and caregiver assessment burden and administration/scoring time. Thus, the aim of the current study was to identify items from the SPSI-R that would provide for a more efficient assessment of global social problem-solving abilities. This study consisted of three independent samples: 121 persons in low-vision rehabilitation ( M age = 71 years old, SD = 15.53), 301 persons living with diabetes mellitus ( M age = 58, and SD = 14.85), and 131 family caregivers of persons with severe disabilities ( M age = 56 years old, SD = 12.15). All persons completed a version of the SPSI-R, Center for Epidemiological Studies Depression Scale (CES-D), and the Satisfaction with Life Scale (SWLS). Using Rasch scaling of the SPSI-R short-form, we identified a subset of 10 items that reflected the five-component model of social problem solving. The 10 items were separately validated on the sample of persons living with diabetes mellitus and the sample of family caregivers of persons with severe disabilities. Results indicate that the efficient 10-item version, analyzed separately for all three samples, demonstrated good reliability and validity characteristics similar to the established SPSI-R short form. The 10-item version of the SPSI-R represents a brief, effective way in which clinicians and researchers in busy health care settings can quickly assess global problem-solving abilities and identify those persons at-risk for complicated adjustment. Implications for the assessment of social problem-solving abilities are discussed.

Social problem-solving abilities have been continuously found to influence adjustment among people in general and among those with a wide variety of emotional and medical health concerns ( Dreer, Elliott, Fletcher, & Swanson, 2005 ; D’Zurilla & Nezu, 2007 ; Elliott, Grant, & Miller, 2004 ; Heppner, Witty, & Dixon, 2004 ; Hills-Briggs et al., 2006 ; Nezu, Nezu, Friedman, & Houts, 1999 ). For example, cross-sectional and prospective research has demonstrated that social problem-solving abilities are correlated in predicted directions with perceptions of health and physical symptoms ( Elliott & Marmarosh, 1994 ), secondary health complications ( Elliott & Shewchuk, 2003 ; Herrick, Elliott, & Crow, 1994 ), health compromising behaviors ( Dreer, Elliott, & Tucker, 2004 ), and adherence to medical regimens ( Johnson, Elliott, Neilands, Morin, & Chesney, 2006 ). Social problem-solving abilities have also been found to influence the adjustment of family members who assume caregiving responsibilities of persons with chronic disabilities ( Bambara, Owsley, Wadley, Martin, Porter, & Dreer, in press ; Dreer, Bambara, Simms, Snow, & Owsley, 2008 ; Elliott & Shewchuk, 2003 ; Elliott, Shewchuk, & Richards, 2001 ; Grant, Elliott, Giger, & Bartolucci, 2001 ; Kurylo, Elliott, DeVivo, & Dreer, 2004 ). In addition to the body of correlational research, evidence has also shown that interventions targeting social problem-solving abilities are effective in minimizing subsequent psychological distress and depression for both persons with chronic health conditions ( Cameron, Shin, Williams, & Stewart, 2004 ; Nezu, Nezu, Friedman, & Faddis, 1998 ; Perri et al., 2001 ) and their family caregivers ( Grant, Elliott, Weaver, Bartolucci, & Giger, 2002 ; Houts, Nezu, Nezu, & Bucher, 1996 ; Noojin & Wallender, 1997 ). Thus, measuring social problem-solving abilities in medical health care settings is important for identifying those who may be at risk for problems throughout rehabilitation and postdischarge. Despite the influence on adjustment, outcomes, and implications for treatment, routine assessment of social problem-solving abilities in medical settings is typically underutilized.

A number of empirically supported measures of social problem solving exist in the literature including the Means-End Problem Solving Procedure (MEPS; Platt & Spivack, 1975 ), the Problem Solving Inventory (PSI; Heppner, 1988 ), and the Social Problem Solving Inventory-Revised (SPSI-R; D’Zurilla, Nezu, & Maydeu-Olivares, 2002 ). Although several of these empirically based measures assess a specific aspect of problem solving, only the SPSI-R provides for a more comprehensive assessment of all theoretical components linked to contemporary models of social problem solving (i.e. problem orientation and problem-solving styles; D’Zurilla & Nezu, 2007 ; Nezu, Nezu, & D’Zurilla, 2007 ). More specifically, the SPSI-R is a theory-based measure of social problem-solving processes and has two current versions: (a) a comprehensive 52-item version, which consists of five scales (NPO—negative problem orientation, PPO—positive problem orientation, RPS—rational problem solving, ICS—impulsivity/carelessness style, AS—avoidance style), four subscales of the RPS scale, and a total global score; and (b) a 25-item short form version, which consists of the same five scales identified above and a total global score. The SPSI-R assesses a person’s perception of his or her general approach and styles toward solving problems in everyday living that are bothersome and has been repeatedly found to be reliable and valid ( Chang, 2002 ; D’Zurilla, Chang, & Sanna, 2003 ; D’Zurilla & Maydeu-Olivares, 1995 ; D’Zurilla et al., 2002 ; Jaffee & D’Zurilla, 2003 ). The measure has also been shown to be related to important measures of psychological distress, well-being, and social competence (i.e. depression, distress, anxiety, health-related behaviors, life satisfaction, optimism, situational coping, aggression, externalizing behaviors; Chang, 2002 ; Dreer et al., 2005 ; Dreer et al., 2004 ; Dreer, Ronan, Ronan, Dush, & Elliott, 2004 ; D’Zurilla & Chang, 1995 ; D’Zurilla et al., 2003 ; Jaffee & D’Zurilla, 2003 ). In clinical settings, the SPSI-R is often used to facilitate treatment planning, identify those at risk for problems with adjustment, provide insight for recommendations regarding patient disposition, treatment options, tracking of treatment skill acquisition, and follow-up evaluations. The instrument is also frequently used as an outcome measure in clinical trials as well as other research endeavors.

Although both versions of the SPSI-R provide detailed information regarding a person’s social problem-solving orientation and abilities, the length and administration of the measure can be quite cumbersome when used in medical settings where there are competing demands for time on both patients and their family members. Finding time to administer the SPSI-R is often complicated by demands for other immediate medical procedures and services (e.g. surgery, neuroimaging, chemotherapy, occupational and physical therapy; Glasgow et al., 2005 ; Strange, Woolf, & Gjeltema, 2002 ). Thus, participation in these medical-related procedures and decreased length of stay due to managed care may result in less time for detailed evaluations of social problem-solving abilities as well as assessment of other important aspects of psychological functioning (i.e. personality, family functioning, neuropsychological functioning). Because the current health care environment promotes quick and efficient evaluation of physical symptoms, this also leaves many health care providers without sufficient time to accurately screen for aspects of psychological functioning themselves ( Baker, Keenan, & Zwischenberger, 2005 ). Another obstacle in medical settings is related to the fact that persons often have difficulty completing full versions of self-report measures due to fatigue following lengthy medical procedures and appointments or because of complications related to the person’s health condition in general (i.e. pain, problems with breathing, headaches). Although the short form version of the SPSI-R requires less time than its 52-item counterpart (10 minutes vs. 15–20 minutes; D’Zurilla et al., 2002 ), it can in some instances require twice as much time to administer in medical settings when a person’s chronic health condition calls for a slight modification of administration procedures or the person is fatigued (i.e. reading items and response scales to persons with visual impairments, problems with motor functioning, or hearing; Dreer et al., 2005 ). Other factors known to preclude the use of routine assessment of social problem-solving abilities in medical and rehabilitation settings include respondent burden, item and instrument redundancy, shrinking resources, reimbursement, and cognitive impairment ( Andersen & Haley, 1997 ; Tang, 2006 ). Thus, clinicians and researchers in these types of settings are often faced with the difficult choice of using brief and efficient psychometrically sound measures or no measures at all to evaluate patients who may be at-risk for problems with adjustment ( Elder et al., 2006 ; Stroud, McKnight, & Jensen, 2004 ).

Some of these same issues also overlap with reasons for why family member social problem-solving abilities are not routinely assessed in these types of settings. Other reasons may stem from the fact that family members (a) are not typically viewed as “the patient,” (b) are often occupied by accompanying their family relative into his or her medical appointment, (c) are busy completing their family member’s insurance and medical paperwork, or (d) are not able to devote additional time to complete an assessment of their own due other commitments (e.g. returning to work or to other family responsibilities; Kurtz, Kurtz, Given, & Given, 2004 ; Moen, Robison, & Dempster-McClain, 1995 ; Quittner, Glueckauf, & Jackson, 1990 ). Lastly, another reason family caregivers are not routinely assessed in these settings might be because health care providers are not aware of the body of research demonstrating the influence of family social problem-solving abilities on both the patient’s and family member’s adjustment and health outcomes.

These challenges have important implications for the measurement of social problem-solving abilities currently assessed by the SPSI-R short form in clinical and research endeavors in medical settings. It may be possible, for example, that a more efficient assessment of elements representative of global problem-solving abilities may be achieved without the full array of items required for the robust and stable assessment of the five original scales, thereby minimizing respondent burden and item redundancy. Alternatively, it is also possible that specific items in the SPSI-R may be more sensitive indicators of global problem-solving than other items.

Therefore, we conducted the present study to identify items on the SPSI-R short form that might more efficiently assess social problem-solving abilities. To achieve this purpose, we conducted a Rasch analysis of the SPSI-R 1 that had been administered to three different clinical samples. Rasch analysis is an effective tool in item analysis, as it can detail the extent to which an item fits an underlying unidimensional measurement model and provides opportunities for identifying (and potentially deleting) redundant or misfitting items. By conducting these analyses on data collected from three distinct samples, we may examine the generalizability and replicability of the results. This latter feature may also reveal potential benefits for a more streamlined, efficient assessment of social problem-solving abilities particularly in health care settings, where competing time, attentional, and service demands routinely undermine assessment procedures.

The goal of this investigation was not to replace the SPSI-R short form, but rather to examine if there was a more efficient and rapid set of items that would maintain the key components identified in the social problem-solving model and yet address the practical needs of clinicians and researchers. This type of assessment would offer several potential benefits including reduced item redundancy of the SPSI-R short form and patient/caregiver assessment burden (i.e. fatigue, frustration, boredom with answering highly similar questions repeatedly), while minimizing scoring time and retaining the ability to identify those who might benefit from more comprehensive testing and potential psychological intervention.

This study consisted of three independent samples: (a) 121 persons in low-vision rehabilitation, (b) 301 persons living with diabetes mellitus, and (c) 131 family caregivers of persons with severe disabilities. The 10 items were separately validated on a sample of persons living with diabetes mellitus and a sample of family caregivers of persons with severe disabilities.

Participants

Low-vision rehabilitation sample.

This sample consisted of 50 men and 71 women ( N = 121) who were admitted to an outpatient low-vision rehabilitation program. Prospective participants were informed about the study by their ophthalmologist or optometrist following their initial eye examination. Interested individuals were contacted on site by a research assistant and/or clinic staff working in the clinic, and informed consent was obtained. Average age of the sample was 71 years old ( SD = 15.53), and average level of education was 12 years ( SD = 3.29). One-hundred six participants were Caucasian (87.6%) and 15 participants were African American (12.4%). Macular degeneration was the most frequent primary referring diagnosis for the sample (59.5%, N = 72), approximately 9.1% ( N = 11) were diagnosed with diabetic retinopathy as a primary diagnosis, 5% ( N = 6) were diagnosed with glaucoma, 24% ( N = 29) were diagnosed with other various vision impairments, and 1.7% were diagnosed with vision impairments of unknown origin ( N = 2).

Diabetes validation sample

This sample consisted of 313 persons with diabetes who were receiving outpatient services from the Diabetes Education Center in Baton Rouge, Louisiana ( Elliott, Shewchuk, Miller, & Richards, 2001 ). Prospective participants identified from the clinic database by clinic administration were mailed a questionnaire containing a cover letter describing the nature of the study, a brief demographic form that did not request any personal identification, a consent form, and the measures of social problem-solving abilities and adjustment. Included in the packet was a letter from the clinic director describing the study. A stamped envelope with a return address was provided along with instructions for returning materials. There was no monetary incentive for returning materials.

Three-hundred thirteen protocols were received. Usable self-report protocols were available from 301 respondents diagnosed with diabetes (105 men and 196 women). Average age for the sample was 58 years old ( SD = 15.92); 215 persons reported type II and 62 reported type I diabetes (36 did not report diabetes type). One hundred twenty-six persons reported at least a high school education or equivalent, and 37 reported less than a high school education, 136 reported greater than a high school education, 9 reported “other,” and 5 were missing. The sample was predominantly Caucasian ( N = 190), 99 participants were African American, 9 were Native American, 1 was Asian Pacific Islander, 1 was Hispanic, 1 reported “other,” and 1 did not report ethnicity.

Caregiver validation sample

This sample consisted of 131 family caregivers of adults with disabilities such as a cerebrovascular accident ( N = 51), traumatic brain injury ( N = 63), cerebral palsy ( N = 15), multiple sclerosis ( N = 1), and spinal cord injury ( N = 1). The sample was comprised of 99 Caucasians, 30 African Americans, and 2 Hispanic individuals with average age being 56 years ( SD = 12.15). The majority of family caregivers were women ( N = 114) and 17 were men. Their level of education ranged from 7 to 24 years, with an average of 13 years of education ( SD = 2.85). Eighty-two family caregivers were unemployed, 39 were employed part-time, and 10 were employed fulltime. Of the sample, 81 were single, 26 were married, 16 were widowed, 5 were separated, and 3 were divorced.

Participants volunteered to participate in a study of an in-home problem-solving training intervention. Some prospective participants were identified by clinic staff upon learning which family member would be assuming responsibility for providing daily assistance and care to a patient following discharge from an inpatient rehabilitation program. Other recruitment strategies included advertisements sent to local churches, newspapers, and other health care providers in the surrounding areas. All interested caregivers were asked to contact the research coordinator to determine eligibility. Qualified caregivers were then scheduled for an appointment by a trained research staff member and asked to participate in the study. Those who agreed to participate provided informed consent and completed a variety of baseline psychosocial measures.

Social problem-solving abilities

Two versions of the Social Problem-Solving Inventory-Revised (SPSI-R; D’Zurilla et al., 2002 ) were used: (a) the SPSI-R short form, and (b) the SPSI-R long version. The short form is a subset of the 25-items contained in the long form. In the instructional set, the respondent is asked to rate each item in terms of the individual’s general approach and styles toward solving problems in everyday living that are bothersome, but that he or she does not know how to immediately make better or stop from being bothersome (not daily hassles or pressures). The respondent is further instructed to think about problems as either something about the respondent (i.e. his or her thoughts, feelings, behavior, health, or appearance), the respondent’s relationships with other individuals (i.e. relatives, friends, boss), or the respondent’s environment and things he or she owns (i.e. property, money).

The SPSI-R 25-item short form assesses two constructive or adaptive social problem-solving dimensions (PPO—positive problem orientation and RPS— rational problem solving) and three dysfunctional dimensions (NPO—negative problem orientation, IPC—impulsive/careless style, and AS—avoidance style). The PPO assesses a general cognitive set, which includes the tendency to view problems in a positive light, to see them as challenges rather than threats, and to be optimistic regarding the existence of a solution and one’s ability to detect and implement effective solutions. In contrast, the NPO assesses a cognitive–emotional set that prevents effective problem solving. The RPS assesses an individual’s tendency to use effective social problem-solving techniques systematically and deliberately, including defining the problem, generating alternatives, evaluating alternatives, and implementing solutions and evaluating outcomes. The ICS evaluates a tendency to implement skills in an impulsive, incomplete, and haphazard manner. The AS measures dysfunctional patterns of social problem solving characterized by putting the problem off and waiting for problems to solve themselves. Higher scores on each factor denote greater intensity on a particular dimension. Items on the SPSI-R are rated on a 5-point Likert-type scale (0 = not at all true of me to 4 = extremely true of me ). Participants indicate how they usually respond to everyday problems. Psychometric properties for the SPSI-R short-from have demonstrated good reliability and validity ( D’Zurilla et al., 2002 ).

The SPSI-R 52-item long form (SPSI-R; D’Zurilla et al., 2002 ) is used to more comprehensively assess social problem-solving abilities. The SPSI-R is based on a five-dimensional model of problem-solving and provides five scales ( D’Zurilla et al., 2002 ) as described above (PPO, RPS, NPO, ICS, & AS) along with four subscales of the RPS. Similar to the SPSI-R short form, the long version has well-documented psychometric properties ( D’Zurilla et al., 2002 ).

In the present study, the SPSI-R short form was administered to the low-vision sample. As part of separate, ongoing studies, the SPSI-R long-form version was administered to both the caregiver sample and the diabetes sample. Although the long form was administered to these samples, in the present study we examined only the subset of 25 items, which constitute the short form because our purpose was to compare the performance of a 10-item version to the existing 25-item short form of the SPSI-R. A composite raw score total for the final 10-item version can be computed by summing all 10 items after reverse scoring the items from the NPO, AS, and ICS subscales. Total scores on the 10-item SPSI-R version can range from 0 to 40, with higher scores reflecting greater effective social problem-solving abilities.

Depressive behavior

The Center for Epidemiological Studies Depression Scale (CES-D; Radloff, 1977 ) provides an index of depressive behavior. Higher scores are indicative of depressive symptomatology.

Life satisfaction

The Satisfaction with Life Scale (SWLS; Diener, Emmons, Larsen, & Griffin, 1985 ) evaluates subjective well-being and overall life satisfaction. The SWLS is a five-item instrument with items rated on a Likert-type response format ranging from 1 ( strongly disagree ) to 7 ( strongly agree ). Higher scores reflect greater subjective well-being. Psychometric studies of the SWLS have evidenced internal consistency (α = 0.87) and reliability (2-month test-retest coefficient = 0.82; Diener et al., 1985 ).

The low-vision rehabilitation sample was administered the SPSI-R short form and several other psychological measures following their initial clinical eye examination. The validation samples were administered the SPSI-R long form. All measures were read aloud to the low-vision and caregiver samples; the diabetes sample completed the measures on their own recognizance and returned the completed questionnaires via mail. The institutional review board approved the study procedures for each study.

Data Analysis

The goal of the study was to identify items from the SPSI-R that might provide an efficient assessment of social problem-solving abilities and that correlated with measures of personal adjustment in a manner consistent with the theoretical model of social problem-solving abilities. Our first aim was to determine whether all items of the 25-item SPSI-R short form could be combined into a usefully unidimensional measure of general problem-solving ability. In previous studies, the items of the SPSI-R (both long and short forms) have been successfully fit to a 5-factor model ( Maydeu-Olivares & D’Zurilla, 1996 ; Siu & Shek, 2005 ; Wakeling, 2007 ). However, D’Zurilla et al. (2002) suggest that an SPSI-R total score can be constructed for assessing general problem-solving functioning. The SPSI-R total scores (from both the long and short versions) have been used in many recent studies (e.g. Bray, Barrowclough, & Lobban, 2007 ; Dolgin et al., 2007 ; Fitzpatrick, Witte, & Schmidt, 2005 ; Golden, Gatchel, & Cahill, 2006 ; Kant, D’Zurilla, & Maydeu-Olivares, 1997 ; McMurran, Egan, Blair, & Richardson, 2001 ). An implicit assumption in combining any set of items into a scale is that the items are at least approximately unidimensional ( Wright & Linacre, 1989 ). Creating a total score from all SPSI-R items is partly justified by the high internal consistencies found for the full set of items, but internal consistency is a weak test of dimensionality ( Hattie, 1985 ). In the present study, we evaluated the dimensionality of the SPSI-R total scale (short form only) by fitting all 25 items to an explicitly unidimensional Rasch model ( Smith & Miao, 1994 ; Stansbury, Ried, & Velozo, 2006 ). Specifically, we used the Rasch rating scale model ( Andrich, 1978 ) to assess the measurement structure of the full 25-item scale. The scaling was conducted using Winsteps (Version 3.48; Linacre, 2006 ). The psychometric properties of the full 25-item scale was assessed separately in all three samples described above, including correlations of the Rasch measures with CES-D and SWLS.

Our second aim in this study was to construct a reduced, 10-item version of the SPSI-R that might be useful as a screening tool for quick assessment of general problem-solving abilities. We wanted the reduced version to reflect the same item content coverage as the full 25-item SPSI-R scale. The 25-item scale includes 5 items for each of five scales. To maintain content validity with the full scale, we selected two items from each scale for inclusion in our reduced 10-item scale. We used the results from the Rasch scaling in the low-vision sample for our initial item selection. Specifically, we selected two items from each of the five SPRI-R scales with the lowest mean-square fit statistics. These 10 items were then fit to the Rasch model to assess item functioning and dimensionality of the reduced scale. The measurement properties of the 10 items were then cross-validated in the other two samples described above. The concurrent validity of the 10-item version was assessed by correlations of Rasch person measures with scales of depression and life satisfaction. (In Rasch scaling, ordinal raw score totals for each person are converted to interval [logit] measures. A raw score-to-logit conversion table for the 10-item version is available upon request from the corresponding author.) In all cross-validation analyses, the performance of the 10-item version was compared to the performance of the SPSI-R short form (25-item) scale.

Rasch Scaling of the SPSI-R Short-Form Items

To verify that the original 25-items of the SPSI-R short form could be combined into a global, unidimensional measure of problem solving, the items were fit to the Rasch model for rating scales in the sample of low-vision patients. The Rasch item separation reliability was 0.98 (values > 0.90 are acceptable), which indicates that the items are sufficiently separated in locations (“difficulties”) to form a measurement continuum. The person separation reliability (similar to Cronbach’s alpha coefficient) was 0.85, indicating that the scale is usefully distinguishing among levels of individual problem-solving effectiveness. The fit of individual items to the measurement model was evaluated using information-weighted mean-square fit statistics, which have an expected value of 1. Items with fit statistics less than 1.5 contribute successfully to the construction of measures ( Linacre, 2003 ). All items fit the Rasch model adequately, with mean squares ranging from .66 to 1.48.

Item Selection for an Efficient Assessment of the SPSI-R: Evaluation of a 10-Item Version

The two best fitting items from each of the five SPRI-R scales were identified, and these 10 items were then fit to the Rasch rating scale model. The item separation reliability was .93; the person separation reliability was 0.72.

In Table 1 , we present the item difficulty estimates and the mean-square fit statistics for each item (under Low-vision rehabilitation). Lower values of item difficulties indicate “easier-to-endorse” items. All items fit the model successfully, with fit statistics ranging from .68 to 1.44. Descriptive statistics for the raw score sum of the 10 items are also presented in Table 1 . The correlation between measures on the 10-item and 25-item versions was .94, p < 0.001.

Item Statistics and Raw Score Total Statistics for the Efficient 10-Item Version of the Social Problem-Solving Inventory-Revised (SPSI-R) by Sample

Note . Items are sorted by item location estimates obtained in the combined samples; item numbers are based on the ordering of the original SPSI-R short form. PPO = Positive problem orientation; RPS = rational problem solving; NPO = negative problem orientation; ICS = impulsivity/carelessness style; AS = avoidance style; d = estimated item location (“difficulty”) in logits; MS Fit = the weighted mean square fit statistic.

In Table 2 (see Low-vision rehabilitation), we present the correlations of the Rasch person measures from the 10-item and 25-item SPSI-R versions with the CES-D and the SWLS. Both social problem-solving versions were correlated negatively with depression and positively with life satisfaction, supporting the concurrent validity of the global problem-solving scores. The magnitudes of the correlations obtained with the 10-item version indicates that the reduction of the global SPSI-R composite score from 25 items to 10 items provided minimal loss of predictive power.

Comparison of Validity Between the SPSI-R Short Form and the Efficient 10-Item SPSI-R Version Among Samples

Note . SPSI-R = Social Problem-Solving Inventory-Revised; CES-D = Center for Epidemiological Studies Depression Scale; SWLS = Satisfaction with Life Scale.

Cross-Validation

The results from the previous analyses were cross-validated in the sample of persons with diabetes and in the sample of family caregivers. In both cross-validation samples, the 25-items of the SPSI-R short form functioned adequately as a measure of global problem-solving effectiveness. Item separation reliabilities were 0.98 and 0.95 for the diabetic and caregiver samples, respectively; the corresponding person separation reliabilities were 0.87 and 0.88. Item fit statistics in the two samples ranged from 0.81 to 1.48. The 10 items were also cross-validated. Item separation reliabilities were .98 and .95 for the diabetic and caregiver samples, respectively; person separation reliabilities were 0.74 and 0.72; and item fit statistics in the two samples ranged from 0.78 to 1.26. Table 1 displays estimates and fit statistics, and also descriptive statistics for the raw score total of the 10 items. The correlations between the Rasch measures from the 10-item and 25-item version in the diabetic and caregiver samples were .93 and .94, respectively.

In Table 2 (see Diabetes and Caregiver samples), we also show the correlations of the Rasch measures of the 10-item and 25-item versions with the CES-D and SWLS. In both cross-validation samples, both the 10-item and 25-item versions were significantly correlated with depression and life satisfaction. As in the initial validation sample, the reduced version of the scale showed no great loss of predictive power in any of the analyses.

Scale Structure Stability

To assess the stability of the item locations in the reduced 10-item version across samples, we correlated item location estimates obtained separately in the validation and cross-validation samples. The correlation between locations in the validation and diabetic sample was 0.65; between the validation and caregiver sample, .78; and between the two cross-validation samples, 0.88. To assess stability in the absolute level of item difficulties, we examined the displacement of item locations from the initial validation sample to the cross-validation samples. The item structure from the validation sample was used to anchor the items in the two cross-validation samples. Displacement is the difference between the anchored item structure and the structure estimated from the current data. An item displacement greater than 1 logit is often taken as an upper limit of acceptability; displacements less than half a logit are negligible ( Linacre, 2006 ). Item displacements for the caregiver sample ranged from −0.63 to 0.53; displacements for the diabetic sample ranged from −0.37 to 0.42. We also assessed uniform differential item functioning (DIF) between the three samples as another check on whether item difficulties were comparable across groups ( Bond & Fox, 2001 ). First, Rasch scaling was conducted on item responses from subjects in all three samples combined, which provided anchor values for subject measures and the rating scale structure. Next, scaling was conducted for each sample separately, using anchor values from the combined analysis to equate the measures to a common scale ( Bond & Fox, 2001 ). The DIF was evaluated by subtracting location estimates obtained in the three samples. Differences of less than half a logit are considered evidence of stability ( Wright & Douglas, 1975 ). The differences among item location estimates ranged from −0.44 to 0.54. Only one item had a somewhat higher value than half a logit. The results from these analyses suggest an adequate degree of measurement structure stability across samples.

In this study, we examined the internal psychometric properties (using Rasch scaling) and concurrent validity of a more efficient 10-item version of the SPSI-R short form to assess social problem-solving abilities. Results from the Rasch analysis suggest that the reliability and item fit for the 10-item version was satisfactory and comparable across samples. Based on Rash analysis, the 10-item and 25-item SPSI-R versions were found to be generally equivalent for these samples. The 10 items retain the overall unidimensional, global score of social problem-solving abilities, which correlates with CES-D and SWLS scores. The 10 items also demonstrate stable psychometric properties and may be substituted for its longer counterparts without loss of predictive power. Based on these results, the items appear to efficiently assess global social problem-solving abilities. Thus, clinicians and researchers might want to consider using this quick and efficient version when administrations of longer yet comprehensive versions of the SPSI-R measures are precluded in medical/rehabilitation settings and populations.

These results have important clinical implications. A brief assessment of global social problem-solving abilities consisting of 10 items may be short enough to serve as a routine and time-efficient measure, while minimizing respondent burden, fatigue, and item redundancy. The brevity of this 10-item SPSI-R version may also increase the likelihood of more routine assessment of social problem-solving abilities, and integrate well into other clinical assessment protocols (including bedside and other inpatient administrations in health care settings). These 10 items can be conveniently administered, scored, and interpreted. These features will increase the likelihood of their use by various health care providers and in settings in which particular chronic health conditions limit respondents from completing forms independently (e.g. visual impairments, loss or restriction of limb functioning, connection to IVs, hearing impairments).

The efficient assessment of social problem-solving abilities in health care settings may be particularly welcome as evidence indicates that patients with dysfunctional problem-solving abilities are more likely to demonstrate problems with distress, adherence, and secondary complications over time (i.e. Elliott, Bush, & Chen, 2006 ; Johnson et al., 2006 ). These individuals may benefit from problem-solving training provided in brief interactions with physicians in primary care settings ( Mynors-Wallis, Gath, Lloyd-Thomas, & Tomlinson, 1995 ), in telephone counseling provided by nurses ( Grant et al., 2002 ), and from formal individual and group therapies conducted by mental health providers ( Perri et al., 2001 ; Rath, Simon, Langenbahn, Sherr, & Diller, 2003 ). Because research suggests that health care professionals, for a variety of reasons, often fail to detect common psychological problems ( Baker et al., 2005 ; Haley et al., 1998 ; Saunders & Wojcik, 2004 ), efficient and accurate screening for patients and their family caregivers may provide health care practitioners with relevant information from which to identify and refer persons for further psychological assessment if needed. Efficient assessment of ineffective social problem-solving abilities may ultimately translate into better emotional and physical adjustment ( Karlawish, Casarett, Klocinski, & Clark, 2001 ) for those at risk for complicated adjustment.

The 10-item SPSI-R assessment may also be beneficial for research purposes as well when the length of the SPSI-R 52-item version or 25-item short form version is not feasible. For instance, the 10-item version may be recommended for populations that have great restrictions in their ability to complete questionnaires such as those persons with significant chronic health conditions, acute and severe psychiatric symptoms, and/or there are significant time demands for other medical procedures. The brevity and lack of redundancy of items may help to maximize the chances of inclusion and minimize attrition in research projects where multiple outcome measures are needed. An efficient assessment may also allow for the incorporation of other measures in studies examining the relationships between multiple constructs and for quickly monitoring responses to clinical interventions (i.e. problem-solving training) with difficult-to-assess medical populations. The minimization of item redundancy may also help to maintain participant interest and attention for completing research protocols that are often lengthy in large-scale clinical trials. However, when time permits, research participants should be given the 25-item form or 52-item given the wealth of psychometric data supporting these versions.

Although there were several strengths of the current investigation, the present study remains a preliminary validation of items obtained from the SPSI-R for alternative social problem-solving assessment. As a result, several limitations and directions for future research exist. First, the current study was limited to persons who were living under conditions of chronic stress. Examination of the 10-item SPSI-R version with other samples would help to confirm the generalizability of the present findings. The present study was cross-sectional in nature; longitudinal investigations evaluating the predictive nature of the identified SPSI-R assessment items are warranted. All measures included in the study were self-report questionnaires. The association between the 10-item version and the criterion measures might be due, in part, to shared method variance (i.e. mail-in/written vs. read aloud). Future studies examining these 10 items of the SPSI-R with social problem-solving performance-based measures are needed. Use of more objective measures of problem-solving and adjustment would provide important information about the sensitivity and usefulness of these items in everyday living. It should also be noted that with a 10-item measure, one loses the ability to assess the five-dimensional structure of problem solving. However, as discussed this may be necessary in health care settings where constraints prohibit a full-scale evaluation. Lastly, another limitation may be due to the variance in method administration (i.e. read aloud to patients with visual impairments vs. written/mail in).

Despite these limitations, the current findings suggest that a reduced amount of items from the SPSI-R may serve as an efficient measure for evaluating general social problem-solving abilities. Although these identified items do not replace the use of the full-length SPSI-R, our data suggest that a brief assessment may be effectively accomplished with the items we have identified.

Acknowledgments

Funded by the grant provided by the National Eye Institute (# 1 K23 EY017327-01, Dreer PI), EyeSight Foundation of Alabama (ESFA, Dreer PI), Research to Prevent Blindness (RPB, Dreer, PI), National Institute on Child Health and Human Development (1R01 HD37661-01A3), the National Institute on Disability and Rehabilitation Research (H133N5009, H133B30025-96A & H133B980016A), and the National Center for Injury Prevention and Control (R49/CCR403641). The contents of this article are solely the responsibility of the authors and do not necessarily represent the official views of the funding agencies.

1 To obtain the 10 items that were selected from the SPSI-R short form for this efficient version of social problem-solving abilities, interested persons must contact the owners of the instrument: Multi-Health Systems, Inc., North Tonawanda, New York. Their contact information is as follows: 1-800-456-3003 (U.S.), 1-800-268-6011 (Canada),416-492-2627 (International); e-mail: moc.shm@ecivresremotsuc ; Web site: www.mhs.com

Contributor Information

Laura E. Dreer, University of Alabama at Birmingham.

Jack Berry, University of Alabama at Birmingham.

Patricia Rivera, University of Alabama at Birmingham.

Marsha Snow, University of Alabama at Birmingham.

Timothy R. Elliott, Texas A & M University.

Doreen Miller, Southern University.

Todd D. Little, University of Kansas.

• Andersen B, Haley W. Clinical geropsychology: Implications for practice in medical settings. Journal of Clinical Psychology in Medical Settings. 1997; 4 :193–205. [ Google Scholar ]
• Andrich D. A rating formulation for ordered response categories. Psychometrika. 1978; 43 :561–573. [ Google Scholar ]
• Baker J, Keenan L, Zwischenberger J. A model for primary care psychology with general thoracic surgical patients. Journal of Clinical Psychology in Medical Settings. 2005; 12 :359–366. [ Google Scholar ]
• Bambara J, Owsley C, Wadley V, Martin R, Porter C, Dreer LE. Family caregiver social problem-solving abilities and adjustment to caring for a relative with vision loss. Investigative Ophthalmology and Visual Science. :50. (in press) [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Bond T, Fox C. Applying the Rasch model: Fundamental measurement in the human sciences. Hillsdale, NJ: Erlbaum; 2001. [ Google Scholar ]
• Bray S, Barrowclough C, Lobban F. The social problem-solving abilities of people with borderline personality disorder. Behaviour Research and Therapy. 2007; 45 :1409–1417. [ PubMed ] [ Google Scholar ]
• Cameron JI, Shin JL, Williams D, Stewart DE. A brief problem-solving intervention for family caregivers to individuals with advanced cancer. Journal of Psychosomatic Research. 2004; 57 :137–143. [ PubMed ] [ Google Scholar ]
• Chang EC. Examining the link between perfectionism and psychological maladjustment: Social problem solving as a buffer. Cognitive Therapy and Research. 2002; 26 :581–595. [ Google Scholar ]
• Diener E, Emmons R, Larsen R, Griffin S. The Satisfaction with Life Scale. Journal of Personality Assessment. 1985; 49 :71–75. [ PubMed ] [ Google Scholar ]
• Dolgin M, Phipps S, Fairclough D, Sahler O, Askins M, Noll R, et al. Trajectories of adjustment in mothers of children with newly diagnosed cancer: A natural history investigation. Journal of Pediatric Psychology. 2007; 32 :771–782. [ PubMed ] [ Google Scholar ]
• Dreer LE, Bambara JK, Simms LB, Snow M, Owsley C. Family member psychosocial adjustment and low vision rehabilitation. Paper presented at the 9th International Conference on Low Vision; Montreal, Canada. 2008. [ Google Scholar ]
• Dreer L, Elliott T, Fletcher D, Swanson M. Social problem-solving abilities and psychological adjustment of persons in low vision rehabilitation. Rehabilitation Psychology. 2005; 50 :232–238. [ Google Scholar ]
• Dreer L, Elliott T, Tucker E. Social problem-solving abilities and health behaviors among persons with a recent-onset spinal cord injury. Journal of Clinical Psychology in Medical Settings. 2004; 11 :7–13. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Dreer L, Ronan G, Ronan D, Dush D, Elliott T. Binge drinking in college students: An investigation of social problem-solving abilities. Journal of College Student Development. 2004; 45 :303–315. [ Google Scholar ]
• D’Zurilla T, Chang E. The relations between social problem solving and coping. Cognitive Therapy and Research. 1995; 19 :547–562. [ Google Scholar ]
• D’Zurilla T, Chang E, Sanna L. Self-esteem and social problem solving as predictors of aggression in college students. Journal of Social & Clinical Psychology. 2003; 22 :424–440. [ Google Scholar ]
• D’Zurilla T, Maydeu-Olivares A. Conceptual and methodological issues in social problem-solving assessment. Behavior Therapy. 1995; 26 :409–432. [ Google Scholar ]
• D’Zurilla T, Nezu A. Problem-solving therapy: A positive approach to clinical intervention. 3rd edn. New York: Springer Publishing Co.; 2007. [ Google Scholar ]
• D’Zurilla T, Nezu A, Maydeu-Olivares A. Social Problem-Solving Inventory-Revised (SPSI-R): Manual. North Tonawanda, NY: Multi-Health Systems; 2002. [ Google Scholar ]
• Elder K, Grilo C, Masheb R, Rothschild B, Burke-Martindale C, Brody M. Comparison of two self-report instruments for assessing binge eating in bariatric surgery candidates. Behaviour Research and Therapy. 2006; 44 :545–560. [ PubMed ] [ Google Scholar ]
• Elliott T, Grant J, Miller D. Social problem solving abilities and behavioral health. In: Chang E, D’Zurilla T, Sanna L, editors. Social problem solving: Theory, research, and training. Washington, DC: American Psychological Association; 2004. [ Google Scholar ]
• Elliott T, Marmarosh C. Problem-solving appraisal, health complaints, and health-related expectancies. Journal of Counseling Development. 1994; 72 :531–537. [ Google Scholar ]
• Elliott T, Shewchuk R. Social problem-solving abilities and distress among family members assuming a caregiving role. British Journal of Health Psychology. 2003; 8 :149–163. [ PubMed ] [ Google Scholar ]
• Elliott T, Shewchuk R, Richards J. Family caregiver social problem-solving abilities and adjustment during the initial year of the caregiving role. Journal of Counseling Psychology. 2001; 48 :223–232. [ Google Scholar ]
• Elliott TR, Bush BA, Chen Y. Social problem-solving abilities predict pressure sore occurrence in the first 3 years of spinal cord injury. Rehabilitation Psychology. 2006; 51 :69–77. [ Google Scholar ]
• Elliott TR, Shewchuk R, Miller D, Richards JS. Profiles in problem solving: Psychological well-being and distress among persons with diabetes mellitus. Journal of Clinical Psychology in Medical Settings. 2001; 8 :283–291. [ Google Scholar ]
• Fitzpatrick K, Witte T, Schmidt N. Randomized controlled trial of a brief problem-orientation intervention for suicide ideation. Behavior Therapy. 2005; 36 :323–333. [ Google Scholar ]
• Glasgow R, Ory M, Klesges L, Cifuentes M, Fernald D, Green L. Practical and relevant self-report measures of patient health behaviors for primary care research. Annals of Family Medicine. 2005; 3 :73–81. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Golden L, Gatchel R, Cahill M. Evaluating the effectiveness of the National Institute of Corrections’ “Thinking for Change” Program among probationers. Journal of Offender Rehabilitation. 2006; 43 :55–73. [ Google Scholar ]
• Grant J, Elliott T, Giger J, Bartolucci A. Social problem-solving abilities, social support, and adjustment among family caregivers of individuals with a stroke. Rehabilitation Psychology. 2001; 46 :44–57. [ Google Scholar ]
• Grant J, Elliott T, Weaver M, Bartolucci A, Giger J. A telephone intervention with family caregivers of stroke survivors after rehabilitation. Stroke. 2002; 33 :2060–2065. [ PubMed ] [ Google Scholar ]
• Haley W, Mcdaniel S, Bray J, Frank R, Heldring M, Bennett-Johnson S, et al. Psychological practice in primary care settings: Practical tips for clinicians. Professional Psychology, Research and Practice. 1998; 29 :237–244. [ Google Scholar ]
• Hattie J. Methodology review: Assessing unidimensionality of tests and items. Applied Psychological Measurement. 1985; 9 :139–164. [ Google Scholar ]
• Heppner P. The Problem-Solving Inventory. Palo Alto, CA: Consulting Psychologist Press; 1988. [ Google Scholar ]
• Heppner PP, Witty TE, Dixon WA. Problem-solving appraisal: Helping normal people lead better lives. Counseling Psychologist. 2004; 32 (3):466–472. [ Google Scholar ]
• Herrick S, Elliott T, Crow F. Self-appraised problem solving skills and the prediction of secondary complications among persons with spinal cord injury. Journal of Clinical Psychology in Medical Settings. 1994; 1 :262–283. [ PubMed ] [ Google Scholar ]
• Hills-Briggs F, Gary TL, Yen H-C, Batts-Turner M, Powe NR, Saudek CD, et al. Association of social problem solving with glycemic control in a sample of urban African Americans with type 2 diabetes. Journal of Behavioral Medicine. 2006; 29 :69–78. [ PubMed ] [ Google Scholar ]
• Houts P, Nezu A, Nezu C, Bucher J. The prepared family caregiver: A problem-solving approach to family caregiver education. Patient Education and Counseling. 1996; 27 :63–73. [ PubMed ] [ Google Scholar ]
• Jaffee WB, D’Zurilla TJ. Adolescent problem solving, parent problem solving, and externalizing behavior in adolescents. Behavior Therapy. 2003; 34 (3):295–311. [ Google Scholar ]
• Johnson MO, Elliott TR, Neilands TB, Morin SF, Chesney MA. A social problem-solving model of adherence to HIV medications. Health Psychology. 2006; 25 :355–363. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Kant G, D’Zurilla T, Maydeu-Olivares A. Social problem solving as a mediator of stress-related depression and anxiety in middle-aged and elderly community residents. Cognitive Therapy and Research. 1997; 21 :73–96. [ Google Scholar ]
• Karlawish J, Casarett D, Klocinski J, Clark C. The relationship between caregivers’ global ratings of Alzheimer’s disease patients’ quality of life, disease severity, and the caregiving experience. Journal of the American Geriatrics Society. 2001; 49 :1066–1070. [ PubMed ] [ Google Scholar ]
• Kurtz M, Kurtz J, Given C, Given B. Depression and physical health among family caregivers of geriatric patients with cancer-a longitudinal view. Medical Science Monitor. 2004; 10 :447–456. [ PubMed ] [ Google Scholar ]
• Kurylo M, Elliott T, DeVivo L, Dreer L. Caregiver social problem-solving abilities and family member adjustment following congestive heart failure. Journal of Clinical Psychology in Medical Settings. 2004; 11 :151–157. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Linacre J. Size vs. significance: Standardized chi-square fit statistic Rash. Measurement Transactions. 2003; 17 :918. [ Google Scholar ]
• Linacre J. A users guide to WINSTEPS Ministep: Rasch-model computer programs. Chicag, IL: University of Chicago; 2006. [ Google Scholar ]
• Maydeu-Olivares A, D’Zurilla T. A factor-analytic study of the Social Problem-Solving Inventory: An integration of theory and data. Cognitive Therapy and Research. 1996; 20 :115–133. [ Google Scholar ]
• McMurran M, Egan V, Blair M, Richardson C. The relationship between social problem-solving and personality in mentally disordered offenders. Personality and Individual Differences. 2001; 30 :517–524. [ Google Scholar ]
• Moen P, Robison J, Dempster-McClain D. Caregiving and women’s well-being: A life course approach. Journal of Health and Social Behavior. 1995; 36 :259–273. [ PubMed ] [ Google Scholar ]
• Mynors-Wallis L, Gath D, Lloyd-Thomas A, Tomlinson D. Randomised controlled trial comparing problem solving treatment with amitriptyline and placebo for major depression in primary care. British Medical Journal. 1995; 310 :441–445. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Nezu A, Nezu C, D’Zurilla T. Solving life’s problems: A 5-step guide to enhanced well-being. Springer Publishing Co.; 2007. [ Google Scholar ]
• Nezu A, Nezu C, Friedman S, Faddis S. Helping cancer patients cope: A problem-solving approach. Washington, DC: American Psychological Association; 1998. [ Google Scholar ]
• Nezu A, Nezu C, Friedman S, Houts P. Cancer and psychological distress: Two investigations regarding the role of social problem-solving. Journal of Psychosocial Oncology. 1999; 16 :27–40. [ Google Scholar ]
• Noojin A, Wallender J. Perceived stress and problem solving ability in mothers of children with physical disabilities: Potential cognitive influences on adjustment. International Journal of Behavioral Medicine. 1997; 4 :415–432. [ PubMed ] [ Google Scholar ]
• Perri M, Nezu A, McKelvey W, Shermer R, Renjilian D, Viegener B. Relapse prevention training and problem-solving therapy in the long-term management of obesity. Journal of Consulting and Clinical Psychology. 2001; 69 :722–726. [ PubMed ] [ Google Scholar ]
• Platt J, Spivack G. Manual for the means-ends problem solving procedures (MEPS): A measure of interpersonal cognitive problem-solving skills. Philadelphia: Hahnemann Community Mental Health Mental Retardation Center; 1975. [ Google Scholar ]
• Quittner A, Glueckauf R, Jackson D. Chronic parenting stress: Moderating versus mediating effects of social support. Journal of Personality and Social Psychology. 1990; 59 :1266–1278. [ PubMed ] [ Google Scholar ]
• Radloff L. The CES-D Scale: A self-report depression scale for research in the general population. Applied Psychological Measurement. 1977; 1 :385–401. [ Google Scholar ]
• Rath J, Simon D, Langenbahn D, Sherr R, Diller L. Group treatment of problem-solving deficits in outpatients with traumatic brain injury: A randomized outcome study. Neuropsychological Rehabilitation. 2003; 13 :461–488. [ Google Scholar ]
• Saunders S, Wojcik J. The reliability and validity of a brief self-report questionnaire to screen for mental health problems: The Health Dynamics Inventory. Journal of Clinical Psychology in Medical Settings. 2004; 11 :233–241. [ Google Scholar ]
• Siu A, Shek D. Relations between social problem solving and indicators of interpersonal and family well-being among Chinese adolescents in Hong Kong. Social Indicators Research. 2005; 71 :517–539. [ Google Scholar ]
• Smith R, Miao C. Assessing unidimensionality for Rasch measurement. In: Wilson M, editor. Objective measurement: Theory into practice. Vol. 2. Greenwich, NJ: Ablex; 1994. pp. 316–327. [ Google Scholar ]
• Stansbury JP, Ried LD, Velozo C. Unidimensionality and bandwidth in the Center for Epidemiologic Studies Depression (CES-D) Scale. Journal of Personality Assessment. 2006; 86 :10–22. [ PubMed ] [ Google Scholar ]
• Strange K, Woolf S, Gjeltema K. One minute for prevention: The power of leveraging to fulfill the promise of health behavior counseling. American Journal of Preventative Medicine. 2002; 22 :320–323. [ PubMed ] [ Google Scholar ]
• Stroud M, McKnight P, Jensen M. Assessment of self-reported physical activity in patients with chronic pain: Development of an abbreviated Roland-Morris Disability Scale. The Journal of Pain. 2004; 5 :257–263. [ PubMed ] [ Google Scholar ]
• Tang S. Concordance of quality of life assessments between terminally ill cancer patients and their primary family caregivers in Taiwan. Cancer Nursing. 2006; 29 :49–57. [ PubMed ] [ Google Scholar ]
• Wakeling H. The psychometric validation of the Social Problem-Solving Inventory-Revised with UK incarcerated sexual offenders. Sex Abuse. 2007; 19 :217–236. [ PubMed ] [ Google Scholar ]
• Wright B, Douglas G. Better procedures for sample-free item analysis. Chicago, IL: University of Chicago, Department of Education Statistical Laboratory; 1975. [ Google Scholar ]
• Wright B, Linacre J. Observations are always ordinal; measurements, however, must be interval. Archives of Physical Medicine and Rehabilitation. 1989; 71 :857–860. [ PubMed ] [ Google Scholar ]

By Audience

• Therapist Toolbox
• Teacher Toolbox
• Parent Toolbox
• Explore All

By Category

• Organization
• Impulse Control
• When Executive Function Skills Impair Handwriting
• Executive Functioning in School
• Executive Functioning Skills- Teach Planning and Prioritization
• Adults With Executive Function Disorder
• How to Teach Foresight
• Bilateral Coordination
• Hand Strengthening Activities
• What is Finger Isolation?
• Occupational Therapy at Home
• Fine Motor Skills Needed at School
• What are Fine Motor Skills
• Fine Motor Activities to Improve Open Thumb Web Space
• Indoor Toddler Activities
• Outdoor Play
• Self-Dressing
• Best Shoe Tying Tips
• Potty Training
• Cooking With Kids
• Scissor Skills
• Line Awareness
• Spatial Awareness
• Size Awareness
• Pencil Control
• Pencil Grasp
• Letter Formation
• Proprioception
• How to Create a Sensory Diet
• Visual Perception
• Eye-Hand Coordination
• How Vision Problems Affect Learning
• Vision Activities for Kids
• What is Visual Attention?
• Activities to Improve Smooth Visual Pursuits
• What is Visual Scanning
• Classroom Accommodations for Visual Impairments

Pipe Cleaner Bunny and Carrots

• Free Resources
• Members Club
• Development , Executive Functioning Skills , Mental Health , Occupational Therapy , Occupational Therapy Activities , Self Regulation

The Size of The Problem Activity Ideas

Colleen beck.

• by Colleen Beck
• March 19, 2024

In this blog post, we’re sharing a fun way to work on problem solving skills and overwhelm in kids. The kids we’ve worked with on executive functioning skills love The Size of the Problem Activity strategies for a few reasons. It helps them to understand just how big various daily problems actually are…so they can come up with a game plan to fix the issue at hand.

The size of the problem activity might sound like a quirky game, but it’s actually a combination of self-regulation , executive function, and metacognition that helps kids understand the magnitude of the problems they encounter and to cope with them….all while knowing that it’s ok to have problems, it’s ok to have big feelings, and it’s ok to not know what to do about them. After all, emotional regulation and executive functioning skills go hand in hand.

One way that I’ve done this in the past is with a few fun and engaging activity ideas. These activities can be used with different ages.

What Is “the Size of the Problem”?

Have you heard of a Size of the Problem concept? It’s a tool to help kids picture how bit their problems are.

The concept of “Size of the Problem” activity is often used in educational and therapeutic settings. It essentially helps the child to categorizing problems into different sizes based on their level of seriousness or impact. 

The goal is to teach individuals, especially kids, to match their reactions to the size of the problem, promoting emotional regulation and effective problem-solving.

Helping kids to identify problems by size can help them with skills like:

• Self regulation
• Impulse control
• Working memory
• Emotional regulation
• Time management
• Planning and prioritization
• Social and emotional skills
• Self reflection
• Emotional Control
• Task Initiation
• Task Completion
• Working Memory
• Mental Dexterity
• Prioritizing
• Processing Speed
• Self control
• Self-Monitoring
• Cognitive Flexibility
• Problem Solving
• Persistence

Remember that this can be a tricky skill to learn and then to use! Executive function develops over a long period of time and identifying problems, finding a regulation strategy, and then using it takes time, too!

How Does the size of the problem activity Work?

Now, let’s take a closer look at how this activity works:

• Identification: Kids are guided to recognize and express their feelings about a particular situation or issue. You can prompt them with questions like “How does this make you feel?” or “Why is this bothering you?” 
• Small problems: Minor issues that can be managed independently.
• Medium problems: Require more effort, support, or coping strategies to address.
• Big problems: Significantly impactful challenges that may necessitate assistance from adults or professionals.
• Understanding Impact: By assigning a size to the problem, kids can better understand how it relates to their overall well-being and differentiate between minor frustrations and more substantial issues.
• Applying Coping Strategies: Once the size of the problem is determined, appropriate coping strategies can be introduced. For instance, small problems may be addressed with simple self-soothing techniques, while bigger problems might require more complex problem-solving skills or external support.

How to Introduce the Size of the Problem to Kids?

Introducing the concept of the Size of the Problem to kids should be done in a simple manner. Here’s how you can make the introduction effective and relatable:

• Start with Simple Language: Begin by explaining the concept in language that suits the age group. Use examples they can relate to, like misplacing a favorite toy, having a disagreement with a friend, or not understanding a homework assignment.
• Use Visual Aids: Create a visual chart or use images to represent problems of varying sizes. This can be a spectrum ranging from small to large, helping kids visualize the different magnitudes of problems.
• Open Communication: Encourage conversation by asking them questions. For example, “Can you think of a time when something bothered you? Was it a small, medium, or big problem?” This prompts reflection and allows the child to connect the concept to their own experiences.
• Relate It to Emotions: Discuss how different-sized problems can make us feel various emotions. Connect emotions like frustration, sadness, or worry to specific examples. This helps children link the concept to their own emotional experiences.
• Model the Concept: Model the behavior by sharing your own experiences and categorizing problems based on their size. This helps children see real-life applications of the concept.

Using the Size of the Problem Activity in Different Settings

So, how does this play out in real life? Whether you’re at home, in school, or in a therapy session, the Size of the Problem activity is adaptable. It’s all part of emotional intelligence , but there are different problems that arise in different settings.

Let’s take a quick look at how it can be used in different settings:

Size of the Problem Scenarios At Home

So, you can help kids to understand that different problems come in different sizes by talking through the everyday problems that come up in the day to day at home.

We’ve all experienced issues that derail our plans, and this is true for sure, at home. For example, just this week in our house, we lost a bathing suit that is needed for swim class. A problem like this means that without her bathing suit, my daughter can’t swim in her swim class at school. She will have to either find her bathing suit in the laundry bins/stuffed in a bag/lost under the bed/etc. or she will sit out in swim class. She would then receive a zero for the day.

On the scale of small/medium/large problems, this one is pretty big because it means she would miss swim class each day until the bathing suit is found.

At home, parents play a huge role in the co-regulation that needs to occur as part of development. We can talk with our kids about mood and affect , emotions, and problem solving.

Other “size of the problem” scenarios that might happen at home include:

You need milk or another ingredient for making a meal. This problem is pretty small because there are easy options to solve the problem (ask a neighbor for a cup of milk, go to the store to get milk, make a different recipe that doesn’t require milk). However, if you need milk because a baby or toddler needs that as their primary source of nutrients, then the problem is bigger. It’s more of an immediate need. Some ways you could talk about this problem to support skill building might be:

• Make a family meal plan so you can see what ingredients are needed. This works on planning and working memory skills.
• Keep a checklist of ingredients that you need to pick up from the store. Getting kids involved with this (they can add items when they see the house has run out, too!). This can help kids with working memory, problem solving, and planning skills.

Everyone got up late and now you’re running late for the school bus. This is a larger problem because it has immediate, significant consequences like being marked tardy for school and then work for the adults. It requires a more urgent and structured response (driving to school). Strategies to address this problem could include:

• Teaching time management skills, such as setting alarms or creating a morning routine checklist.
• Problem-solving skills to identify what caused the delay and how to prevent it in the future.
• Emotional regulation skills to manage the stress or anxiety that might arise from running late.

You are unable to find the remote control. This is a smaller problem. It may cause frustration or inconvenience (especially when a favorite show is on tv) but lacks significant or long-lasting consequences. The approach to this issue is more about managing disappointment or frustration and finding creative solutions. This small problem is actually a great way to teach skills to our kids, that they can use for other problems.

• Encouraging the child to express their feelings in a constructive manner, and use self regulation strategies .
• Teaching organizational skills or systematic ways to look for lost items.
• Highlighting the difference in the scale of reactions appropriate for small problems versus big problems.

The list could go on and on (and on)! Problems are part of day to day life, because nothing is exactly like we might predict it to be. But, as parents, we can use these problems to help our kids develop real and essential skills.

Some ways to talk about and come up with tools to “go with the flow” when problems arise at home (and they will):

• Family Discussions: Gather the family and initiate casual discussions about daily experiences. Ask questions like, “What happened today that made you happy, and was there anything that bothered you?” Encourage kids to share and categorize problems based on their size.
• Visual Aids: Create a visible chart or poster at home depicting the Size of the Problem spectrum. Include pictures or symbols to represent different-sized problems. This serves as a daily reminder and facilitates ongoing conversations about emotions.
• Family Coping Strategies: Introduce and practice coping strategies as a family. Emphasize that everyone has different ways of dealing with problems, and it’s okay to seek support from one another. Make it a collaborative effort to build a positive and supportive home environment.

Size of the Problem Scenarios In School

You can probably see that identifying problem size and coping with that problem is actually a life skill. It makes sense that as parents we can help our kids develop these skill and that the home is a great place to work on them.

But, we all know that problems will arise at school too! You can even include some of these concepts and ideas in a calm down area in the school. For example, using an emotions check in  activity or a  feelings check in activities  can help with this ability.

For example, some ways that size of the problem activities can be done at school include:

The student is missing a school assignment. This is a larger problem in the school context because it has direct consequences on the child’s grades. Things do come up, though so missing assignments are not always going to be a big issue, and it’s up to the teacher to decide on that. How big of a problem it is might depend on if the student consistently misses assignments, or other considerations.

This type of problem also provides an important learning opportunity about responsibility and time management.

Addressing this problem could involve:

• Helping the child understand the importance of deadlines and how missing them can impact their grades.
• Developing time management and organizational skills, such as using a planner or setting reminders.
• Working with the child to communicate with the teacher about the missed deadline and to understand the consequences and responsibility.

The student has a disagreement with a friend at recess. This is a smaller problem (in most cases), with less long-term consequences, but it’s an important opportunity to develop social skills .

Addressing this issue can involve:

• Teaching the child to express their feelings and listen to others’ perspectives, fostering empathy and communication skills.
• Encouraging problem-solving strategies to resolve disagreements, such as finding a compromise or seeking help from a teacher or peer mediator.
• Highlighting the importance of resilience and the ability to bounce back from minor social conflicts.

Some ways to help address various size of the problem scenarios at school include:

• Visual Aids in Classrooms: Teachers can display visual aids representing the Size of the Problem spectrum in the classroom.
• Role-Playing Exercises: Classroom activities can include role-playing exercises where students act out scenarios and categorize problems. This hands-on approach fosters a deeper understanding of the concept and encourages peer-to-peer discussions.
• Classroom Coping Strategies: Integrate coping strategies into the classroom routine. You can also incorporate self-regulation strategies . Teach students various coping mechanisms and encourage them to apply these strategies based on the size of the problem they encounter.

Size of the Problem activities

We covered some size of the problem strategies in the scenarios above, and these ideas can be applied to a bunch of different situations.

Now, let’s look at some strategies that align with the Size of the Problem activities. The idea is to match the intensity of the coping strategy with the size of the problem. 

These can be great self regulation group activities for a small group in schools.

We do have a few printable resources that can be used:

• For small, medium, or large problems, you can also help students to use a goal ladder to help them identify steps they need to take to reach their goals when it comes to problems.
• Another tool is our resource to help kids break down goals .
• Another printable resource is our drawing mind map exercises . You can use them to help kids figure out different responses and what to do about problems.

The thing to remember is that problems mean responses. What seems like a small problem to one person might actually be a huge problem to the person actually experiencing it! And that’s totally ok! I like to think about it like the Zones of Regulation where it’s ok to not be in the green zone all of the time. It’s OK to feel emotions and have big feelings to things like losing the remote!

As therapy providers, and as parents and educators, it’s actually our job to not argue about how much a child should be responding to a problem, but to accept those feelings and then to offer solutions. Maybe some ideas for what to do next, or what to do next time can help!

One way to do this is with sorting problems, much like our measuring activities , only in this case, we’re helping kids to measure out the size of an issue they might be experiencing.

Small Problems

Small problems can use different tools that support small needs. A student can use these ideas to help.

• Breathing Exercises: Teach deep breathing exercises for small problems. A few mindful breaths can bring a sense of calm and perspective.
• Using a Stress Ball or Fidget Toy: Provide a small stress-relief tool. Squeezing a stress ball or using a fidget toy can be a quick and effective way to release tension.
• Taking a Short Break: Suggest a short break from the situation. Sometimes stepping away briefly can reset their emotions for small problems.

Medium Problems

• Journaling: Introduce journaling as a coping strategy . Writing about their feelings and thoughts can help kids process medium-sized challenges.
• Problem-Solving Techniques: Teach basic problem-solving skills. Guide them in breaking down the issue into smaller parts and brainstorming possible solutions.
• Positive Self-Talk: Encourage positive self-talk. Help children develop phrases like “I can handle this” or “It’s just a small bump in the road” for minor issues.

Large Problems

• Seek Adult Guidance: Encourage reaching out to trusted adults. For larger problems, seeking guidance from parents, teachers, or counselors is an appropriate and essential step.
• Create a Plan: Work together to create a plan. Break down the larger problem into manageable steps, helping kids feel more in control.
• Professional Support : Emphasize the importance of professional support. For significant challenges, seeking help from a therapist or counselor can provide the necessary tools and guidance.

Size of the Problem Books

One way to help kids with problem solving and identifying what problems they are having…and then what to do about it…is with books. Some of the ones that I’ve used in the past (and love) include:

• The Problem with Problems
• Of COURSE It’s a Big Deal
• Gloria’s Big Problem
• Barnie’s Little Big Problem
• A Very Big Problem
• Big Problems Little Problems
• What Do You Do With a Problem?
• Solutions for Cold Feet and Other Little Problems
• My Day is Ruined!
• Don’t Squeal Unless it’s a Big Deal

When reading these books with kids, you can help them to pay attention to the problems and what the characters in the books did about them.

How Can you help kids with size of the problem

Hopefully, these ideas gave you something to think about. The important thing to take from these ideas is that identifying the size of a problem isn’t meant to dismiss feelings we have about a problem. It’s actually normal to “feel” no matter what the problem’s size is. Helping kids to identify a problem by size is simply a tool that shapes the way kids understand and handle their emotions.

It helps them to use the regulation strategies that work for them. And it helps them work through those emotions.

I like that we can support kids, no matter what emotions they feel about a specific problem, and give them tools to meet those needs.

So, if you are a therapy provider working on social and emotional regulation skills with kids, know that the curriculum isn’t always cut and dry. That part comes with skilled therapy experience. We can equip our kiddos with the skills needed to assess, understand, and effectively manage challenges. This is part of function!

Colleen Beck, OTR/L has been an occupational therapist since 2000, working in school-based, hand therapy, outpatient peds, EI, and SNF. Colleen created The OT Toolbox to inspire therapists, teachers, and parents with easy and fun tools to help children thrive. Read her story about going from an OT making $3/hour (after paying for kids’ childcare) to a full-time OT resource creator for millions of readers. Want to collaborate? Send an email to [email protected].

More Posts Like This

• Crafts , Fine Motor Skills , Occupational Therapy

• Fine Motor Skills , Occupational Therapy , Sensory

Oral Motor Exercises with a Cotton Ball Bunny

• Occupational Therapy , Occupational Therapy Activities

Easter Activities

• Executive Functioning Skills , Occupational Therapy Activities , Sensory

Coping Strategies for Kids

Quick links, sign up for the ot toolbox newsletter.

Get the latest tools and resources sent right to your inbox!

Get Connected

• Want to read the website AD-FREE?
• Want to access all of our downloads in one place?
• Want done for you therapy tools and materials

Join The OT Toolbox Member’s Club!