I AM YUEYIN JI
CLC case study: CASE 2
Abstract
In assessing three potential curricula for the SkillsAdvantage program, a Quasi experiment was conducted across various districts. Using a difference-in-differences analysis, we compared pre- and post-program scores across control and treatment groups. Program A showed improvements in reading and math, Program B significantly enhanced mathematical skills, and Program C regressed in all areas. The data suggest adopting Program B, with its marked improvement in critical mathematical competencies, despite modest gains in reading and writing.
Introduction
Context:
Three new educational programs have been shortlisted as potential replacements for the current Skills curriculum. To evaluate their efficacy, a test market experiment was designed and conducted across the districts we serve, with each district exclusively using one of the new curricula for a cohort of incoming students. This approach aimed to isolate the impact of each educational program on student performance and program outcomes.
Objectives:
The primary objective of this report is to analyze the data collected from the SkillsAdvantage Curriculum experiment to determine the most effective curriculum for our students.
Specific goals include:
-
Evaluate the performance outcomes of the cohorts under each new educational program relative to the existing curriculum.
-
Offer insights into the suitability of the districts included in the experiment and advise on the representativeness of these districts for broader implementation.
-
Formulate a recommendation on which educational program should be adopted as the new standard for the SkillsAdvantage course.
Experimental Methodology
â‘ Experimental design
Experimental units: educational program group (A, B, C, Current)
Level of treatment: for each group, keep track of their intake test scores before they started the skills program, and their test scores after the skills program.
The control group is the group that takes the current skills program.
Assignment: non-randomized
â‘¡Data design
The dependent variables are students’ reading, writing, math without calculators, math with calculators, and total scores.
Dependent variable:
The suitability of applying DiD methods rests on Parallel trend assumptions. If the assumption is satisfied, we need to conclude that: For students in the SkillsAdvantage program, if they are enrolled in the current skills program, we expect them to raise their scores by the same amount as if they were enrolled in the current skills program.
To verify this assumption: First, we screened out the students who enrolled at Cobblestone's Skills program for their first course. Then, we calculated how much the students improved after the skills program. Based on the student's district and their progress score, we have two ways to verify the assumption.
Method 1：ANOVA test
Judging from the results, the average improvement scores in district 'Wilsonville' differed the most from the average improvement scores in other districts (Wilsonville-Beaverton:15.3426973, Wilsonville-Camas 17.5541581, Wilsonville-Ridgefield:17.4473496, Wilsonville-Sherwood: 18.7223069, Wilsonville-St. Paul:16.8673724), and the p-value was statistically significant.
Method 2: Regression
Holding other things constant, district “Wilsonville” has a 15.3427 higher improved score than District “Beaverton”(estimate coefficient = 15.3427, p < 0.001), while other districts’ improved scores are lower than District ‘Beaverton’. It indicates that the variation of students' progress scores in district "Wilsonville" is larger than that in other districts, which indicates that district "Wilsonville" does not meet the assumption.
As a result, we should not include district "Wilsonville" in the analysis.
â‘¢Analysis design
Since the experiment is non-randomized, we should use a diff-in-diff analysis.
The following is the framework of DiD analysis designed by us
Results & Discussion
I took the current program group as the baseline and compared the scores and improved scores in each project after the skills program with those of other programs. The results are shown in the visualization below.
-
Diff in Diff Graphically:
Compared to the current program, program A has a positive impact on student’s score.
Compared to the current program, program B has a positive impact on student’s score.
​​Compared to the current program, program C has a negative impact on student’s score.
Results Visualization
This is a comparative view of the performance in reading and writing and mathematical skills. Each skill's performance is indicated by bars in different colors representing each program.
Program B is the most promising option, especially in enhancing mathematical skills without a calculator. Program A provides balanced enhancement across skills except for a slight decline in math without a calculator. The improvement effect of Program B and A on writing skills is not as good as other skills. Program C shows an overall decrease in skills, particularly in mathematical areas.
This is an aggregated view of the total changes across all tested skills for each new program compared to the current program's baseline. The 'total' bar at the bottom shows the numerical values of the aggregate changes for each program.
Compared to the current group, program A exhibits a modest overall improvement. Program B displays a significant overall improvement, the highest among the three programs. Program C shows a notable overall decline in performance.
Conclusion
-
Main Findings
Program A demonstrated consistent improvements across reading and mathematical calculations, with a slight improvement in writing. Program B, while showing modest improvements in reading and writing, made significant strides in mathematical skills, both with and without calculators. Conversely, Program C resulted in a regression in all tested skills.
​
2. Research Question Answered
The research aimed to identify which new educational program most effectively enhances student performance in the SkillsAdvantage course. The findings answer this question, indicating that Program B outperforms the other programs in terms of overall score improvements, particularly in mathematics.
​
3. Recommendation
Given the results, it is recommended that the client consider adopting Program B for the SkillsAdvantage course. The substantial improvements in mathematical outcomes suggest that this program could provide students with robust skills that are likely to be beneficial in the job market.At the same time, companies should pay more attention to the teaching of writing skills, because the new program does not significantly help students improve their writing ability compared with other abilities.Moreover, the company should further investigate the reasons behind Program C's negative impact on student scores to ensure these pitfalls are avoided in future curriculum development or implementation strategies.