Statement of the Hypothesis

The Claims

Responding to student achievement data published by Stanford Education Data Archive (SEDA)1, a New York Times article of April 29, 20162 addresses the question, “Why racial achievement gaps were so pronounced in affluent school districts is a puzzling question raised by the data.” In particular, the article looks at the Chapel Hill-Carrboro school district. In an interview, the school district presents the following statement. “The wealthier students tend to come from families where, “let’s face it, both the parents are Ph.D.s, and that kid, no matter what happens in the school, is pressured from kindergarten to succeed” … “So even though our minority students are outscoring minority students in other districts near us, there is still a bigger gap here because of that.” The context of the article is the online Educational Opportunity Project at Stanford University3. The academic parent of the Stanford site is the Stanford Center for Education Policy Analysis (CEPA)4.

The school district’s statement appears to be composed of multiple claims:

  1. Chapel Hill-Carrboro (CHCCS) White students come from economically well-off families;
  2. CHCCS White students are high achievers;
  3. CHCCS minority students outscore minority students in nearby districts;
  4. The minority-White GLP Gaps in CHCCS are more, that is, worse, compared to the students in those other districts;
  5. The minority-White GLP Gaps are a consequence of 1) and 2) and do not infer lesser minority-White equity compared to other North Carolina districts.

There is, though, a substantial problem with the somewhat oblique question to which the school district is responding, and accordingly with the response itself. Whatever the origin, I pose here an alternative question which I will also address in this report: Given that there are school districts where White students perform at very high levels, to what extent do minority students participate in that excellence?

This report will carry out an investigation of these claims in the context of the North Carolina Department of Public Instruction (NCDPI) data made available publicly in their disaggregated data files5. My analysis will be different from that of CEPA in that I will be looking at the Grade Level Proficiency (GLP) measure reported, as a percentage, by DPI. In grades 3 through 8, this is determined from standardized grade-appropriate examinations, including mathematics and language arts, and additionally science for grade 8. The DPI “Green Books”6 describe test results. The Green Books provide detailed, statewide data for some grades and some subjects as score-frequency tables for test results. From these, as shown in other reports, it is evident that in some grades, GLP is achieved by reaching the median state-wide test score.

In regards to the score-frequency data, those tests are not graded “on the curve.” Rather, they are carefully designed according to the discipline known as Item Response Methodology. These tests have a series of four steps or threshholds, i.e., five divisions, that require the demonstration of specific abilities, and they are vetted prior to statewide use.78 This is discussed in the 2015 NCDPI report on alignment characteristics of assessments.9 To score into the third or higher division, i.e., being at least GLP, students would have to demonstrate capabilities beyond what are required of those who remain in the first or second division. This changed somewhat in 2019, when Levels 1 and 2 (the lowest achievement clasifications) for mathematics were collapsed into a single level.10

Looking at a map of North Carolina, the Local Education Agencies (LEA) nearby to Chapel Hill-Carrboro include Orange County, Chatham County, Alamance County, Caswell County, Person County, and Durham County. Wake County is separated from Orange County but is part of the Chapel Hill-Durham-Wake metropolitan area. This report’s comparisons will treat first the nearby districts, 010 Alamance-Burlington, 170 Caswell County, 190 Chatham County, 320 Durham, 680 Orange County, 681 Chapel Hill-Carrboro City, 730 Person County, and then the more economically comparable districts,111 Asheville City, 190 Chatham County, 290 Davidson County, 320 Durham, 600 Charlotte-Mecklenburg, 650 New Hanover County, 681 Chapel Hill-Carrboro City, 920 Wake County. Appendix A provides a brief characterization of the nearby and peer LEAs.

The NYT article makes reference to minority students. Instead of “race,” DPI uses the term “subgroup,” being composed of American Indian, Asian, Black, Hispanic, Multiracial, Pacific Islander, and White.11 Due to the small numbers of some of the subgroups and consequent difficulties in comparability, this report will consider only Black, Hispanic, and White students. Other reports in this series will also address Asian students.

I do not include charter schools, which warrant a separate treatment since over the years they vary in number of students served and grades offered, making longitudinal comparisons difficult.

The Black-White and the Hispanic-White GLP Gaps within the Chapel Hill-Carrboro LEA

To establish a baseline, I first consider the DPI disaggregated data for the years from 2013-14 through 2018-19 for the Chapel Hill-Carrboro schools. Figure 1 shows the values of GLP percent for White, Black, and the White-Black difference. This is aggregated for each school and includes all students in all grades. It is evident that change in the Black-White gap is determined primarily by change in the Black GLP, not by change in the White GLP which is quite similar in value year to year.

The plots in Figures 2 use the same data as in Figure 1, but show the individual CHCCS schools. For clarity, Figures 2 show the GLP percentages, but not the gaps. The detail that emerges in these plots shows that aggregating at the LEA level hides a great deal of information.

It is useful to look at the CHCCS GLP scores in a way that incorporates the variations among the individual schools. Figure 3 shows the aggregate scores as box and whiskers plots. The median scores are the horizontal lines in the boxes, and the means are indicated by X. It can be seen that there are small variations in the scores for White students, while those for the Black and Hispanic students show considerable variability.

Comparison with Nearby Districts

The nearby districts, i.e., the LEAs, used in this report are 010 Alamance-Burlington, 170 Caswell County, 190 Chatham County, 320 Durham, 680 Orange County, 681 Chapel Hill-Carrboro City, 730 Person County. A comparison of these LEAs is shown in the next table. This includes all traitional schools, elementary, middle and high schools.

LEA District nALL nASIA nBLCK nHISP nWHTE nOTHR pctASIA pctBLCK pctHISP pctWHTE pctOTHR
010 Alamance-Burlington Schools 22698 348 5125 5988 10088 1149 1.5 22.6 26.4 44.4 5.1
170 Caswell County Schools 2613 7 948 206 1337 115 0.3 36.3 7.9 51.2 4.4
190 Chatham County Schools 8767 104 1010 2765 4478 410 1.2 11.5 31.5 51.1 4.7
320 Durham Schools 33072 744 14804 10130 6226 1168 2.2 44.8 30.6 18.8 3.5
680 Orange County Schools 7319 91 1039 1724 4104 361 1.2 14.2 23.6 56.1 4.9
681 Chapel Hill-Carrboro City Schools 12269 1765 1372 2012 6288 832 14.4 11.2 16.4 51.3 6.8
730 Person County Schools 4436 16 1549 414 2206 251 0.4 34.9 9.3 49.7 5.7

The aggregated GLP and gaps for the nearby districts are in Appendix B. Figure 4 compares, in summary, the GLP for Chapel Hill-Carrboro and the nearby districts.

Black, Hispanic and White Percentage GLP for the Nearby Schools

The following plots, Figure 5, show the Percentage GLP for Black, Hispanic, and White students for each of the nearby LEAs.

Figure 6 is another way to look at the same data that is in Figure 5. It shows the Black White, and the Hispanic White GLP Gaps for the nearby LEAs.

Comparison with Peer Districts

The peer districts, i.e., the LEAs, are 111 Asheville City, 190 Chatham County, 290 Davidson County, 320 Durham, 600 Charlotte-Mecklenburg, 650 New Hanover County, 681 Chapel Hill-Carrboro City, 920 Wake County. The following table presents data for all traditional schools, elementary, middle, and high schools.

LEA District nALL nASIA nBLCK nHISP nWHTE nOTHR pctASIA pctBLCK pctHISP pctWHTE pctOTHR
111 Asheville City Schools 4421 47 899 363 2790 322 1.1 20.3 8.2 63.1 7.3
190 Chatham County Schools 8767 104 1010 2765 4478 410 1.2 11.5 31.5 51.1 4.7
290 Davidson County Schools 19089 224 696 1800 15766 603 1.2 3.6 9.4 82.6 3.2
320 Durham Schools 33072 744 14804 10130 6226 1168 2.2 44.8 30.6 18.8 3.5
600 Charlotte-Mecklenburg Schools 147359 9761 56204 35534 41315 4545 6.6 38.1 24.1 28.0 3.1
650 New Hanover County Schools 26213 468 5017 3580 15946 1202 1.8 19.1 13.7 60.8 4.6
681 Chapel Hill-Carrboro City Schools 12269 1765 1372 2012 6288 832 14.4 11.2 16.4 51.3 6.8
920 Wake County Schools 160095 14160 36765 28329 74241 6600 8.8 23.0 17.7 46.4 4.1

Figure 7 compares, in summary, the GLP for Chapel Hill-Carrboro and the peer districts. The detailed aggregated GLP and gaps by year for the peer districts are in Appendix C.

Black, Hispanic and White Percentage GLP for the Peer District Schools

The following plots, Figure 8, show the GLP for Black, Hispanic, White, and All students for each of the peer LEAs.

Remember, LARGER GLP is better.

Black White and Hispanic White Gaps for the Peer District Schools

Figure 9 shows the Black White, and the Hispanic White GLP Gaps for the peer LEAs.

Remember, SMALLER Gap is better. Note that Wake (920) has smaller Gaps than some other LEAs but that is because for Wake the White Percentage GLP is lower than some other LEAs even though the Black and Hispanic Percentage GLP are quite the same.

Observations and Conclusions

Before responding to the five claims, I turn to the usefulness of the GLP Gap as a measure of equity. The GLP Gap is arrived at by taking the difference between the percent GLP for White students and that for minority students. It is highly reductive, ignoring any indication of the spreads in the percentage GLP of the constituent schools, as well as gathering together all grades. This makes it difficult at best, if not misleading, to compare LEAs for the same year, or to assess changes across years in any specific LEA.

As the difference of two percentages, the GLP Gap is the subject of a further deficiency. It can be made smaller, or even reversed in sign, by decreasing the GLP of the White students. This, indeed, is what is happening in some of the poorer, nearby LEAs. The inference is that equity is achieved when everyone does poorly, which is hardly a desireable outcome. There is, in addition, the problem that comparing percentages is not helpful when at least one of the percentages is high, above eighty or ninety percent, as it is “crowded” against the one-hundred percent limit. Statisticians use a modified measure, called log-odds, to handle this, but in recognition of its complexity I do not use it here.

I call attention to an aspect of the assessment results, namely the longitudinal variability of test scores. Another report treats grades 3, 4, and 5 in detail, using the class-level data in the NCDPI disaggregated data files. My analysis shows large year to year variations, and consequently casts further doubt on the usefulness of the reductive GLP Gap measure.

I have determined that the CHCCS White students achieve consistently higher GLP percentages than do White students in the nearby districts, although Durham (LEA 320) is closer to CHCCS than the other districts. Black students in CHCCS perform somewhat better than Black students in the nearby districts; while the margin is small it is consistent both for the mean and the median. A consequence of the large White percentage GLP is that the CHCCS Black White GLP Gap is apt to be larger than those for the nearby districts. This confirms the school district’s claims in the NYT article. However, aside from Durham, the nearby districts are not as well off as CHCCS, and their schools are not as well funded. Comparing their schools to those of CHCCS is problematic at best, and offers little insight into issues of equity.

Comparison with districts that are more similar to CHCCS, the peer LEAs, leads to a more complicated situation. Comparing Figure 5 with Figure 8, it is evident that the CHCCS White student GLP tends to be marginally higher than that for the other districts. However, in contrast with the nearby districts, the CHCCS Black students performance is no better than typical. Put another way, the Black CHCCS students, aggregated for all grades and subjects, achieve more than do Black students in the economically challenged nearby LEAs, but this does not hold true when they are compared with Black students in the peer LEAs.

Figures 4 and 7 both show that CHCCS White students, aggregated over all grades for all schools, have consistently higher Percentage GLP than do White students in the nearby and peer LEAs. These Figures also show that the CHCCS Percentage GLP for Black and Hispanic students are merely typical.

It follows that in the Chapel Hill-Carrboro school district minority students do not appear to participate in the excellence exhibited by the White students.

It is my conclusion that the Percentage GLP Gap is not a useful measure of equity.

In regard to the five claims, I make the following conclusions:
1) Verified.
2) Verified.
3) Conditionally verified in that minority students do not do better when compared with selected LEAs that are economically better off than the nearby LEAs.
4) and 5) The truth value of these two claims is irrelevant since they are based on the GLP Gap, a faulty measure, and they have the same comparison problem as does claim 3.

Appendix A. Comparison of Selected Economic Measures in Nearby and Peer LEAs

The nearby and peer LEAs differ substantially in wealth. The following school funding data is from the Census Bureau 2017 Public Elementary-Secondary Education Finance Data12. A review of this data shows that total revenue per student (TOTALREV/ENROLL), total local revenue per student (TLOCREV/ENROLL), total expenditure per student (TOTALEXP/ENROLL), and total current spending per student (TCURELSC/ENROLL) have sufficient variability to suggest looking further. It is evident that CHCCS is a leader in per pupil expenditures, with Orange County (LEA 680) and Ashevile City Schools (LEA 111) similar in some but not all four categories.

Fig. A.1,

Fig. A.1,

Per Pupil Amounts in Dollars By Total Local Spending Per Pupil
LEA TREV TLOC TEXP TCUR Which
681 12518 6520 13086 12085 Both
111 12664 5810 13011 11284 Peers
680 12004 5125 12708 11073 Nearby
190 10909 4288 10958 10001 Both
320 11297 4237 11350 10440 Both
650 10178 3330 11815 9750 Peers
600 9794 3300 10674 8998 Peers
730 10804 3103 10412 9750 Nearby
290 9639 3078 10156 8208 Peers
920 8978 2826 11149 8597 Peers
010 8817 2077 8846 8448 Nearby
170 10418 1960 10080 9667 Nearby

I note that, while my choice of peer LEAs includes the comparable Asheville, Durham, and Charlotte-Mecklenburg, it also includes LEAs with expenditures more comparable with the nearby LEAs.

Other characterizations of North Carolina counties (not LEAs) are available from the Census Bureau Quick Facts13, and from the American Community Survey14. I here treat the measures % Population Aged 5 Years and Over Language Other than English Spoken in the Home (pctOtherLangPop); % Households with a Computer (pctHsldWithComp); % Households with Wifi (broadband) (pctHsldWithWifi); % Population High School Graduate or Higher (pctHSplus); % Population with Bachelor’s Degree or Higher Aged 25 Years+ (pctBSplus); % Population Under 65 Years Without Health Insurance (pctNoHins); % Persons in Poverty (pctPov); % Persons Under 18 in Poverty (pctUnder18Pov); Median household income (medHshldInc); and Mean household income (meanHshldInc).

Fig. A.2, Percentage of Population LEA Measures

Fig. A.2, Percentage of Population LEA Measures

Fig. A.3, Median and Mean LEA Income

Fig. A.3, Median and Mean LEA Income

Appendix B. Aggregated Gaps for the Nearby LEAs

LEA f_acYear ALL BLCK HISP WHTE BW_gap HW_gap
010 A 49.3 31.5 37.5 61.5 30.0 24.0
010 B 49.9 32.4 38.9 62.4 30.0 23.5
010 C 51.8 34.1 42.6 64.1 30.0 21.5
010 D 52.5 34.7 44.0 65.3 30.6 21.3
010 E 50.8 33.9 41.8 64.3 30.4 22.5
010 F 51.5 35.0 43.7 65.1 30.1 21.4
170 A 45.6 32.4 54.5 52.9 20.5 -1.6
170 B 43.8 32.9 42.6 50.8 17.9 8.2
170 C 44.0 30.5 38.8 53.5 23.0 14.7
170 D 43.7 32.8 34.1 52.4 19.6 18.3
170 E 43.8 32.2 37.1 52.3 20.1 15.2
170 F 44.0 32.7 40.1 51.9 19.2 11.8
190 A 57.6 34.3 39.5 72.2 37.9 32.7
190 B 57.0 34.3 38.1 72.2 37.9 34.1
190 C 57.0 33.1 40.2 72.1 39.0 31.9
190 D 61.2 36.6 46.3 75.0 38.4 28.7
190 E 61.8 38.3 46.9 76.0 37.7 29.1
190 F 62.4 38.0 49.6 75.5 37.5 25.9
320 A 44.1 34.2 36.5 76.0 41.8 39.5
320 B 44.0 34.2 36.1 77.0 42.8 40.9
320 C 44.9 34.8 37.1 78.6 43.8 41.5
320 D 46.4 36.4 38.3 79.9 43.5 41.6
320 E 48.3 39.3 40.6 80.4 41.1 39.8
320 F 49.0 39.5 41.5 80.9 41.4 39.4
680 A 59.6 34.7 41.2 70.4 35.7 29.2
680 B 59.4 34.7 42.5 70.3 35.6 27.8
680 C 62.2 38.0 46.6 73.1 35.1 26.5
680 D 63.4 39.8 45.4 76.1 36.3 30.7
680 E 61.8 40.5 45.6 73.9 33.4 28.3
680 F 59.4 38.0 43.3 72.5 34.5 29.2
681 A 77.1 39.7 47.9 90.2 50.5 42.3
681 B 76.9 41.9 47.4 90.2 48.3 42.8
681 C 76.6 40.9 48.4 89.9 49.0 41.5
681 D 76.6 39.0 49.9 89.8 50.8 39.9
681 E 74.2 37.2 46.6 89.3 52.1 42.7
681 F 75.5 42.8 49.4 89.6 46.8 40.2
730 A 54.1 39.0 48.6 64.6 25.6 16.0
730 B 53.0 37.1 48.5 63.6 26.5 15.1
730 C 53.8 39.4 49.3 63.6 24.2 14.3
730 D 57.5 42.0 53.5 68.2 26.2 14.7
730 E 55.5 38.8 52.9 65.9 27.1 13.0
730 F 53.1 36.6 50.0 64.7 28.1 14.7

Appendix C. Aggregated Gaps for the Peer LEAs

LEA f_acYear ALL BLCK HISP WHTE BW_gap HW_gap
111 A 66.6 30.0 57.9 83.4 53.4 25.5
111 B 65.2 26.6 54.7 82.5 55.9 27.8
111 C 66.7 29.5 57.0 83.9 54.4 26.9
111 D 65.8 24.5 57.4 82.7 58.2 25.3
111 E 64.2 19.3 55.5 80.9 61.6 25.4
111 F 64.0 20.9 53.1 80.1 59.2 27.0
190 A 57.6 34.3 39.5 72.2 37.9 32.7
190 B 57.0 34.3 38.1 72.2 37.9 34.1
190 C 57.0 33.1 40.2 72.1 39.0 31.9
190 D 61.2 36.6 46.3 75.0 38.4 28.7
190 E 61.8 38.3 46.9 76.0 37.7 29.1
190 F 62.4 38.0 49.6 75.5 37.5 25.9
290 A 60.6 46.5 48.2 62.3 15.8 14.1
290 B 59.9 41.6 47.3 61.9 20.3 14.6
290 C 62.5 45.6 53.1 64.3 18.7 11.2
290 D 63.1 47.2 52.6 65.0 17.8 12.4
290 E 60.8 43.7 51.0 62.8 19.1 11.8
290 F 62.3 45.9 53.3 64.3 18.4 11.0
320 A 44.1 34.2 36.5 76.0 41.8 39.5
320 B 44.0 34.2 36.1 77.0 42.8 40.9
320 C 44.9 34.8 37.1 78.6 43.8 41.5
320 D 46.4 36.4 38.3 79.9 43.5 41.6
320 E 48.3 39.3 40.6 80.4 41.1 39.8
320 F 49.0 39.5 41.5 80.9 41.4 39.4
600 A 59.2 43.7 48.8 82.9 39.2 34.1
600 B 59.4 43.9 48.2 84.1 40.2 35.9
600 C 61.2 46.4 50.5 85.2 38.8 34.7
600 D 61.2 46.6 50.3 85.3 38.7 35.0
600 E 59.9 45.8 49.2 84.4 38.6 35.2
600 F 59.8 46.4 49.0 84.4 38.0 35.4
650 A 64.1 34.2 53.1 76.1 41.9 23.0
650 B 63.3 32.8 52.1 75.6 42.8 23.5
650 C 64.4 33.7 52.8 76.7 43.0 23.9
650 D 64.2 31.8 51.0 77.3 45.5 26.3
650 E 66.2 34.9 52.4 79.2 44.3 26.8
650 F 65.0 35.5 49.6 77.4 41.9 27.8
681 A 77.1 39.7 47.9 90.2 50.5 42.3
681 B 76.9 41.9 47.4 90.2 48.3 42.8
681 C 76.6 40.9 48.4 89.9 49.0 41.5
681 D 76.6 39.0 49.9 89.8 50.8 39.9
681 E 74.2 37.2 46.6 89.3 52.1 42.7
681 F 75.5 42.8 49.4 89.6 46.8 40.2
920 A 66.6 42.3 47.7 81.5 39.2 33.8
920 B 66.8 42.7 47.5 81.8 39.1 34.3
920 C 67.9 44.1 48.7 82.7 38.6 34.0
920 D 67.2 42.9 46.5 82.8 39.9 36.3
920 E 65.7 41.7 44.9 82.1 40.4 37.2
920 F 65.2 42.1 44.7 80.9 38.8 36.2

Supplementary Bibliography as of December 10, 2019

Census Bureau county population estimates via https://www.census.gov/data/tables/time-series/demo/popest/2010s-counties-detail.html

Census Bureau SAIPE data modeling description: https://www.census.gov/data/datasets/time-series/demo/saipe/model-tables.html
SAIPE 2017 data: https://www.census.gov/data/datasets/2017/demo/saipe/2017-state-and-county.html

Census Bureau SAIPE Quantifying Relative Error in the School District Estimates https://www.census.gov/programs-surveys/saipe/guidance/district-estimates.html

NCDPI Free and Reduces Price Meals Eligibility: https://www.dpi.nc.gov/news/press-releases/2019/07/12/eligibility-requirements-set-school-lunch-program

Reardon, S. and Galindo, C., The Hispanic-White Achievement Gap in Math and Reading in Elementary Grades. Amer. Ed. Res. Journ., September 2009, V. 46, No. 3, pp. 853-891.

Reardon, S., Kalogrides, D., Shores, K. The Geography of Racial/Ethnic Test Score Gaps. Stanford Center for Education Policy Analysis (CEDA) Working Paper No. 16-10: https://cepa.stanford.edu/sites/default/files/wp16-10-v201604.pdf

References as of December 10, 2019

  1. Stanford Education Data Archive (SEDA) data for the NYT article: https://purl.stanford.edu/db586ns4974↩︎

  2. The New York Times article: https://www.nytimes.com/interactive/2016/04/29/upshot/money-race-and-success-how-your-school-district-compares.html↩︎

  3. Educational Opportunity Project at Stanford University https://edopportunity.org/↩︎

  4. Stanford Center for Education Policy Analysis: https://cepa.stanford.edu/↩︎

  5. NCDPI disaggregated data files: https://www.dpi.nc.gov/districts-schools/testing-and-school-accountability/school-accountability-and-reporting/accountability-data-sets-and-reports#reports-of-supplemental-disaggregated-state,-school-system-and-school-performance-data↩︎

  6. NCDPI Green Books: https://www.dpi.nc.gov/districts-schools/testing-and-school-accountability/school-accountability-and-reporting/state-testing-results-green-book-archive↩︎

  7. NCDPI End-of-Grade (EOG) Tests https://www.dpi.nc.gov/districts-schools/testing-and-school-accountability/state-tests/end-grade-eog↩︎

  8. North Carolina End-of-Grade Tests of English Language Arts (ELA)/Reading Grades 3-8 (2019) https://www.dpi.nc.gov/documents/achievement-level-ranges-and-descriptors-reading-eog-edition-4↩︎

  9. Smithson, J.L., A Report to the NCDPI On the Alignment Characteristics of State Assessment Instruments Covering Grades 3-8, and High School Mathematics, Reading and Science (2015) https://files.nc.gov/dpi/documents/accountability/testing/technotes/alignreport15.pdf↩︎

  10. NCDPI Achievement Level Ranges for Mathematics (2019) https://files.nc.gov/dpi/documents/files/achievement-level-ranges-and-alds-gen-math-eog.pdf↩︎

  11. Pew Research Center, Lopez, M.H., et al., Who is Hispanic?, (November 11, 2019) https://www.pewresearch.org/fact-tank/2019/11/11/who-is-hispanic/↩︎

  12. Census Bureau 2017 ACS school expenditures file:
    data https://www.census.gov/data/tables/2017/econ/school-finances/secondary-education-finance.html
    item description https://www2.census.gov/programs-surveys/school-finances/tables/2017/secondary-education-finance/school17doc.doc↩︎

  13. Census Bureau Quick Facts via DATA.GOV: https://catalog.data.gov/dataset/quickfacts↩︎

  14. Census Bureau American Community Survey made conveniently available by proximityone http://proximityone.com/sd17dp3.htm↩︎