Prior to FY 2017, all Ohio standardized ("high stakes") tests were paper-based. Starting in FY15, Ohio school districts were given the opportunity of administering these tests by computer. Starting with FY17 (this year), Ohio became a "full-computer" state; all students must now take these tests using a computer online, with only those with health-related or emotional-behavioral disabilities preventing them from taking a computer test permitted to take a paper test. This is legislated in Section 610.17 of HB 64 the 131st General Assembly.
In March of 2016, after the 2015 Report Cards were released, Mr. Michael Molnar of Amherst Exempted Schools surveyed Ohio districts to find out which had administered the 2015 tests by paper and which had administered them by computer. Based on the results of the survey and using ODE Value Added Gain Index Grade data, he evaluated that those districts which had used computer-based testing (CBT) had received an inferior grade.
ODE claimed that there was no difference. (http://education.ohio.gov/Topics/Testing/State-Test-Updates/March-2016/Comparability-of-Paper-and-Online-Testing-Frequent)
In January of this year, we took his survey data and analyzed the impact of CBT on both achievement and progress measures as provided on the Ohio Report Cards.
The results were conclusive. The gap between computer-administered and paper-administered tests was 8 percentage points on the "% Proficient or Better" measure and 6 Value Added Gain Index points for reading, and 2.5 percentage points for % Proficient or Better and 1.5 points for Value Added Gain Index for math. These are significant.
The implications for this are significant.
1. For Students. Clearly the computer-based tests do not afford students the best opportunity to demonstrate their mastery of the material being tested, as computer skills and knowledge play an inordinately influential role in the testing regimen. The tests are particularly important for those students subject to the 3rd grade reading guarantee and graduation requirements.
2. For Teachers. For those districts with pay-for-performance contracts, this provides a complication in determining administrator and teacher compensation.
3. For Students with IEPs. Accommodations (including the administration of paper-based tests) must be documented in the IEP. The opening date for requesting paper tests is February 27. Many Intervention Specialists, IEP teams, and parents are not aware that including paper-based testing is an option for these students.
4. For District Management. This gap and the resulting disconnect make year-over-year comparisons of district performance by management and administrators difficult.
The study provides recommendations for ODE to address the situation.
The full report and presentation to the State Board of Education on 2/14/2017 can be downloaded here. The gist of the study is in Exhibit V.
Furthermore, the impact on a district's performance as reported on its Report Card can be seen in the following graph of the Value Added Gain Index for a district which switched from paper in 2014 to computer in 2015.
Dispatch article on CBT. Click here to read.
This gap existed for all reading and math tests administered for 3rd grade through 8th grade, with the exception of 3rd grade reading and 7th grade math.
For example, for 4th grade reading, below are plots of district measures for computer and paper test results. Clearly paper scores (in blue) were superior to computer scores (in red).
Survey Presentation to Ohio Board of Education
We recently (3//2017) conducted a survey of all Ohio school districts regarding their experience in transitioning to and implementing computer-based testing.
Click here to download the report.
Click on the video below (BOE 3-14-2017) to hear the presentation.