About The Data

The sources of data are the publicly available “disaggregated” files provided by NCDPI on the achievement of students during the academic years from 2013-14 to 2018-19, and publicly available ancillary NCDPI data.[NCDPIdis][NCDPIacc] While this data is extensive, it does not allow following individual students from year to year. In North Carolina the term Local Education Agency (LEA) describes what in other places might be called a school district. Most LEAs match to a county, but some counties have urban areas with their own LEA.[NPIDBlea] The publicly available NCDPI data is compliant with requirements of the federal Family Educational Rights and Privacy Act (FERPA), consequently test scores and numbers of students are reported by grade, not individual class or teacher, for each school.[DOEferpa][REAR2015] Since FERPA requires the masking of some detailed data, I will look at the reporting of Grade Level Proficiency, which is determined by students scoring in the upper three levels of these five-level standardized tests. This avoids much of the masked data, although it makes detailed analysis of the individual levels difficult.[FRYE2005][JACO2016a]

The NCDPI disaggregated data files are organized in categories (called subgroups) including race, gender, the self-reported category of economically disadvantaged students (EDS), and some other categories.[DPIeds] More specifically, race (which is subsumed into subgroup) includes the self-reported categories American Indian, Black, Hispanic, Pacific Islander, and Two or More Races, as well as other categories that are not the subject of this report. Hispanic is treated similarly to Black, White, etc., in that students are not Hispanic and Black, or Hispanic and White, but only one of these.[PEW2019] There is no way to identify students as, for example, Black and EDS, Hispanic and not EDS, etc. A breakdown of school enrollment by race is available at the NCDPI website.[NCDPIenrol]

Part I. North Carolina as Two States

I.1. Wealth and Ethnicity

I begin with the use of the NCDPI data in order to take advantage of what is sometimes called the law of large numbers, also expressed as the central limit theorem and convergence to the mean. This encapsulates the usefulness of the mean of collective data when the mean, or the shape, of each component of the collection might be substantially different. For the present purposes, I look at the means and quantiles of GLP percent across the state and also for individual LEAs and schools. A standard way of presenting this as percentiles is by using Tukey box plots. In these plots, the lower edge of the box is the 25th percentile, the upper edge the 75th percentile, and the horizontal line inside the box is the 50th percentile (the median). The Interquartile Distance (IQD) is the 75th percentile minus the 25th percentile. The upper whisker extends 1.5 times the IQD above the 75th percentile; the lower whisker extends 1.5 times the IQD below the 25th percentile. Points above or below the extent of the whiskers are referred to as outliers. This does not mean that these points are in some way dubious, just that they lie far from the median.

Since I am dealing with GLP percentages which can be sensitive to shifts in small numbers of students if the class sizes are small, I exclude all classes of size less than twenty students. That is, if the total number of students for a particular grade and subject for a school is less than twenty, that specific “class” is excluded. That amounts to 729 of the 25855 grade 3, 4, and 5 classes, associated with 127 of a total of 1589 schools over the 6 years. This approach is found throughout the research literature.[needs reference]

The following figures present a substantiation of the partitioning of North Carolinas assertion.

Figure I.1. GLP percent by Group for Reading in 2018-19

Figure I.1 selects one year (2018-19) and one subject (Reading) across all NC schools, and presents GLP percent box plots for grades 3, 4, and 5. The two left-most boxes for each grade are the EDS and Not-EDS GLP percent, while the three right-most boxes show GLP percent for Black, Hispanic, and White students. Observations for other years and for mathematics are substantially the same. In response to the data masking required by compliance with FERPA, in this report GLP percent above 95% is always shown as 97.5%, while GLP percent less than 5% is shown as 2.5%. This results in some lumpiness at the top and bottom of the GLP percent axis.

I have shown the EDS/Not-EDS categories even though there is extensive data, as well as common knowledge, that White students constitute the preponderance of the Not-EDS category. Presenting these categories along with those for Black, Hispanic, and White, serves to underline the characterization of North Carolina as two states.

Figure I.1 shows that the GLP percent scores for EDS, Black, and Hispanic students are consistently below those for Not-EDS (i.e., wealthier) and White students. Note particularly that the upper bounds of the boxes (the 75th percentile) for the EDS, Black, and Hispanic students are below the lower bounds (the 25th percentile) for the White students. While this is not an argument for statistical significance, it is a strong and consistent indicator of difference.

Figure I.2 selects one grade (5) and one subject (Reading) across the academic years from 2013-14 to 2018-19. This addresses the consistency of the observations based on Figure I.1, showing that the GLP percent data does not change much across the years. However, important aspects of the GLP percentages are obscured by this plot, which I will discuss in Section II of this report.

Figure I.2. GLP percent for Grade 5 Reading Across the Years


Figure I.3 tracks a wave, starting with Reading for grade 3 in 2016-17 and following it to grade 5 in 2018-19. While individual students are not identified or followed, the grade 3 students for each school would tend to move to grade 4, etc. The poorer achievement of EDS, Black, and Hispanic students is consistent across the years. Plots run for other years, as well as for Mathematics, appear much the same.

Figure I.3. GLP percent Reading for Cohort 2016-17 through 2018-19 by Group


From these graphics I observe again the separation between EDS categories, and between White students and Black, as well as Hispanic, students.

I.2. GLP percent Distributions

What I observe here is central to the ability to measure change and to make recommendations to educators and policy makers. Figure I.4 shows the GLP percent profiles for the races Black, Hispanic, and White, and also for the EDS/Not-EDS categories. Note that students belong to one subgroup at a time, so it is not possible to identify students as Black and EDS, White and Hispanic, and so on. The density charts are for GLP percent for 2018-19 Reading for grade 5 over all schools. Other grades, and mathematics, appear similar to Figure I.4.

Figure I.4. 2018-19 Reading for Grade 5

It is evident that the distributions for White and for the wealthier, Not-EDS, students differ structurally from those for the other categories. Not only are the means clearly different, but the shapes are different. The vertical bar just below 100% in Not-EDS and White is an artifact of FERPA compliance and should be imagined as spread out over the 95% to 100% interval. The ALL data is reasonably described as a symmetrical distribution, while the White and Not-EDS lean toward higher percentages; this is called skewness. The Black and EDS distributions have tails into higher GLP percent, while the mass of the students lies in the lower GLP percents. The profiles are so different that it raises the important question of whether the tests are better at identifying the subgroup or assessing performance.[HO2015] This also infers that while there is a component of assessing educational attainment, there is also a component of the ability to take these examinations.

I.3. Wealth Indicators?

What can be said about the influence of economic well-being on student achievement? One quick way to look at that question is to select some economic indicator and compare LEAs. I use the 2017 LEA household income as estimated in the Census Bureau ACS data.[CBind][CBacs][NCESest][PROX] Figures I.5 are ‘slow motion’ versions of Figure I.3, where a statewide quasi-cohort moves from grade 3 in 2016-17 to grade 5 in 2018-19. Each plot shows one grade, with the left-hand boxes applying to LEAs where the median household income was in the lowest twenty percent (the 20th percentile) of the statewide median household incomes. The right-hand boxes are for the wealthier LEAs, those above the 80th median household income percentile.

The Figure I.5 plots show that White and Not-EDS students perform significantly better in the right-hand, wealthier LEAs, while there is much less benefit for the Black, Hispanic, and EDS students.

Do Black and Hispanic Students Perform Better In Wealthier LEAs?
Left-hand boxes are LEAs in 20th Percentile of Median Household Income. Right-hand boxes are LEAs above 80th Percentile.

Figure I.5 Grades 3, 4 and 5

Figure I.5 Grades 3, 4 and 5

I.4. Some Further Aspects of Equity

I treat here some matters of equity by looking at the GLP percent for Black students compared to that for White students. Following are two interactive scatter plots for grades 3, 4, and 5 in the year 2018-19. In the interest of brevity, I treat the cumulative grades 3, 4, and 5 GLP percent across all North Carolina schools. Placing the cursor on a point displays the school code. Schools above the diagonal line have GLP percent for Black students exceeding that for White students. This plot includes what might be a surprisingly small number of schools. Indeed, there were 655 schools with either no Black or no White students in the accumulated grades 3, 4, and 5 in 2018-19. These schools are not part of Figure I.6.

IMPORTANT NOTE: The smaller the count of students, the less dependable is the GLP percent. That is, since we are looking at percentages, GLP percent for student counts of under twenty or so can be substantially affected if only one or two students were to move from the GLP to not GLP, or vice versa. I have not included error bars in the scatterplots because this would make them cluttered and difficult to read.

Figure I.6. Comparing Black and White GLP percent for 2018-19

The interactive scatter plot provides the opportunity to pursue further the best and poorest performing schools.

Figure I.7 is similar to Figure I.6, but shows a comparison of EDS and Not-EDS students. The same striking dichotomy persists.

Figure I.7. Comparing EDS and Not-EDS GLP percent for 2018-19

Figure I.8 addresses a follow-up question: how do Black students perform in the better performing predominantly White schools. In this plot the schools are limited to those where 1) the White GLP percent was at least 75% and 2) there were more White than Black students. There does not appear to be any correlation between the scores of Black and White students. Put another way, Black students do not appear to consistently benefit from being in high-performing, majority White schools, the GLP percent of the Black students being as likely high as low. Factors that contribute to high performance for Black (and for EDS) students are beyond a simplistic attribution to a high White proportion. The interactive scatter plot can be used to identify the schools in which the Black students were best and poorest performing.

Figure I.8. Black Students GLP percent in High-Performing White Schools

Part II. The Nature of the Grade 3, 4, 5 Tests

II.1. Score-Frequency Tables

The extent of aggregation in the publicly available NCDPI data in response to FERPA requirements makes it difficult to use this data to fully understand the characteristics of the tests. However, it is possible to make some useful observations. In particular, the NCDPI Green Books provide score-frequency tables for the tests from 2013-14 to 2018-19. These are for each subject, but are only for the total number of students, not for each subgroup.

There is, of course, a different test for each year for each grade, and for each of the two subjects, Reading and Math. The tests are constructed by educators and psychometricians on the basis of Item Response Theory. This is a complex methodology, different from the more familiar classical test theory.[FAN1998][JACO2016b] The tests are used to classify students into five levels. Students testing into any of the three highest levels are considered to be grade level proficient (GLP).[SMIT2015] If students test into the two highest levels they are considered to be career and college ready (CCR). For the grade 3, 4 and 5 Reading and Math tests, the weighted median score lies at the start of Level 3, that is, within GLP but not in CCR.

Figure II.1 is a visualization of a score-frequency table. This is one of the thirty-six tests covering the 2013-14 to 2018-19 years. While all the tests are similar, there are significant differences in their details. An extensive exposition of the available score-frequency data is presented in Score Frequency Appendix.

Figure II.1. A Representative Score-Frequency Table

Salient characteristics of the tests include the small range of scores, about 60 points of the maximum 470, that students score into. Similar scoring s are used in many tests including those for higher grades, such as the SAT. Nevertheless, the range is 60 points, not 470. It is also evident that there is a step, or discontinuity, that ‘guards’ level 3 (the start of GLP). Level 3 is narrow, only two or three points. The score-frequency tables for all students indicates that Level 3 does not appear to be a ‘trap’ of consequence, since the counts of students in level 4 (CCR) are high. However, my analysis shows otherwise when only Black or Hispanic students are considered.

This report run 2021-05-08