Alumni Impact Study > Methodology
The University of Maine Margaret Chase Smith Policy Center designed this survey in collaboration with the University of Maine Alumni Association, and implemented it between January and April of 2020.
Surveys were sent via email to 38,980 alumni for whom the University of Maine Alumni Association had email addresses. Of those for whom no email address was on file, a random sample of 10,000 received an invitation to participate via mail. At the close of the survey in April, the survey had garnered 5,228 responses. The response rate was about 14% and is similar to the response rates seen in recent MIT and Stanford alumni surveys.
To infer the impact of the entire UMaine alumni population requires “scaling”[1] the observations in the sample. Each sample observation implicitly represents some number of people in the relevant population. If a sample of 1,000 observations comes from a population of 10,000 people, then each observation implicitly represents 10 people. Scaling the observations is assigning the number of people that each observation represents. Although simple in principle, scaling is potentially complicated and problematic for two reasons.
First, the population size is often not known precisely. The number of living UMaine alumni is unknown. Thus, it is not immediately obvious how many alumni are represented by the 5,228 survey responses.
Second, the sample is often not completely representative of its population. The responses to the survey of UMaine alumni may be disproportionately high or low in various dimensions. This could be due to some subgroups of alumni being more/less likely to be contacted for the survey and/or being more/less likely to complete the survey request. For these reasons not all of the survey observations should have the same scaling/weighting factor.
Previous studies for MIT and Stanford dealt with the second issue by estimating different scaling factors across gender-college-decade subgroups.[2] Their analyses found that their survey response rates depended the most on gender, college within the university, and decade of attendance. Thus, they calculated roughly 100 scaling factors for each gender-college-decade cell. Each scaling factor is computed as the inverse of the response rate for each cell.
This study follows the general MIT/Stanford approach. Based on their UMaine major, alumni are grouped into seven “colleges”:[3] College of Education and Human Development; College of Engineering; College of Liberal Arts and Sciences; College of Natural Sciences, Forestry, and Agriculture; Graduate School; Maine Business School; and Other. The “Other” college contains various subgroups that were individually too small to draw reliable inferences on their scaling/weighting factors. It includes UMaine alumni not graduating with a degree, obtaining an associate’s degree or certificate, obtaining a bachelor’s degree in the former University College, etc.
Alumni are also grouped into six decades from the 1960s through the 2010s. Everyone completing their education before 1970 is placed into the 1960s decade. The numbers of students in the 1940s and 1950s in some colleges, such as women in engineering and business, are far too few to draw reliable inferences about their scaling factors. Thus, this report computes inverse response rates for 84 gender-college-decade subgroups (2 × 7 × 6).
The previous studies for MIT and Stanford did not address the first issue above. Evidently, those institutions had contact information for essentially all of their living alumni. If this is indeed the case, their alumni databases can be used to infer their population sizes, and their scaling is as straightforward as noted above. But this is not the case for UMaine.
The UMaine Alumni Association has contact information for 93,177 alumni, but this is probably only about 85% of the living alumni population.[4] This database suggests that each of the 5,228 survey responses represents about 17.8 people, on average. But this understates the appropriate average scaling factor for the number of UMaine alumni and their economic and community impacts. Moreover, in this instance the inverse response rates computed from the alumni database are not necessarily representative of the population subgroups. That is, some subgroups of alumni might be more/less likely to be contacted for a survey request, hence creating an additional possibility that the unweighted sample is not representative of all alumni.
Thus, the inverse response rates for the 84 gender-college-decade subgroups computed from the UMAA database only serves as a starting point for scaling the observations. To infer the total population of alumni, the UMAA database is compared to UMaine records on degrees awarded (from the Office of Institutional Research and Assessment). Since not all recipients of degrees awarded decades ago are still alive, the degree data are adjusted downward based on average death rates. The latest average death rates are reported in the most recent life tables for American women and men from the National Center for Health Statistics.[5] The degree data do not have the ages of the recipients, though. Hence, it assumed that graduation ages are for “traditional” full-time students. That is, bachelors students are assumed to graduate at age 22, and graduate students at age 26.[6] This produces an estimated population of UMaine degrees, but there are several complicating factors.
First, UMaine records on degrees awarded only go back to 1985, but there are living UMaine alumni (and survey completers) going back to the 1940s. There are UMaine Library records, however, that report enrollment by college year going back to the 1940 academic year. That is, there are data on enrollment of seniors (and juniors, sophomores, etc.) that can be used to proxy degrees awarded that academic year. Examination of data on enrollment and degrees after 1985 suggests that senior enrollment is nearly identical to degrees awarded in earlier decades.
Second, college graduates have significantly longer life expectancies than average Americans.[7] Thus, the estimated population of UMaine degrees based on average death rates is on the low side. This is partially offset by some students obtaining degrees later than the traditional ages. Moreover, this report uses life tables for white American women and men because their death rates are noticeably lower than for Americans overall (and also because Maine has relatively little racial diversity). Nonetheless, the estimated population of UMaine degrees appears to be on the conservatively low side.
Despite this, the estimated number of “live” degrees awarded by UMaine since 1985 is 13.0% greater than the number of degrees reported in the UMAA database. Since 1985, UMaine has awarded 70,196 degrees (not including the relatively small number placed in the “Other” college category). The actuarial adjustment suggests that 67,864 of these degrees are held by living alumni. But the UMAA database indicates that its members possessed 60,070 degrees (not including those in the Other category) from UMaine since 1985.
The extent that degrees awarded by UMaine exceeds the number of degrees in the UMAA database was roughly constant since 1985. It is also essentially the same for women and men. But it varies noticeably across colleges. Graduates from the College of Engineering, the College of Natural Sciences, Forestry, and Agriculture, and the Maine Business School are relatively more likely to be in the UMAA database (the average yearly underestimate of degrees is, respectively, 8.9%, 9.3%, and 9.9%), while those from the College of Education and Human Development and the Graduate School are less likely to be included in the UMAA database (17.8% and 23.4%, respectively). Thus, the inverse response rates for the 84 gender-college-decade subgroups computed from the UMAA database are adjusted upwards by the estimated average yearly underestimation percentage for each college.
The third complicating factor is that there are no UMaine records on the numbers of students not eventually earning degrees. Thus, the inverse response rates for the Other college category are not adjusted upward as for the other colleges, and the scaling factors for this category err on the conservatively low side.[8]
Fourth, the comparison of the UMAA database and UMaine degree records are for degrees, not people. Numerous alumni have earned more than one degree from UMaine. In these instances of multiple UMaine degrees the scaling factor for each alumnus/alumna is their average scaling factors for each of their degrees.
Finally, the last complication arises because response rates are not independent of having multiple UMaine degrees. Alumni with multiple degrees are more likely to complete the survey, on average. The UMAA database indicates 1.128 degrees per alumni, but those completing the survey have 1.190 degrees per alumni. Hence, the above comparison of degrees awarded by UMaine to degrees in the UMAA database is 5.5% too low [(1.128-1.190)/1.128]. The scaling factors/weights are thus adjusted accordingly.
When appropriate the scaling factors are adjusted upward to offset instances of unanswered questions within the survey. For example, if 2% of the survey respondents did not answer a question, then to correctly calculate the estimated total effect for that question each response for the 98% who did answer needs to be scaled up by an additional 2%.
Income
All comparison data for annual income are from the 2018 (the latest available) Public Use Microdata Sample of the American Community Survey (ACS) conducted by the U.S. Census Bureau.[9] It has detailed information for nearly 2.4 million adults at least 22 years of age.[10] More than 10,000 of these observations are from Maine residents.
To maintain comparability, instances of top-coded values in the ACS are applied to the survey data (for example, annual employee wages in Maine are top-coded at $356,000; and this is applied in the survey data). Instances of categorical data in the survey data are applied to the ACS to create an apples-to-apples comparison.
Employment
The survey was conducted between January and April of 2020. This means that a portion of survey responses came after the March 13, 2020 Declaration of a National Emergency concerning the novel coronavirus (COVID-19) outbreak.
About fifteen percent of our respondents completed the survey on, or after, this national emergency declaration. For most of the questions in our survey which targeted economic information (income, earnings, or businesses status, etc.) respondents were asked to provide information for 2018 (to enable comparison to national datasets). However, our question about employment asked respondents about their ‘current’ employment status.
Employment figures throughout the report include results from both before and after March 13. UMaine alumni labor force participation and unemployment rates are compared to estimates calculated from the 2020 Current Population Survey (January through March). The first quarter (January through March) Bureau of Labor Statistics data on employment by industry and occupation at the state level were not available at the time of this report. Therefore, we used 2019 Bureau of Labor Statistics Quarterly and Annual Industry Employment and Wages data and 2019 Occupational Employment Statistics Program Occupational Employment and Wage Estimates data to arrive at a general estimate of the proportion of UMaine alumni working in various industries and occupations in Maine. This indicates that our results on the share of alumni working across all industries and occupations in Maine are conservative, as our data on alumni include some information on the employment of alumni after the lockdowns began, and we are comparing to datasets (from 2019) which do not capture this tumultuous time. Our results should be taken as conservative estimates of the magnitude of UMaine alumni working across various industries and occupations in Maine prior to the coronavirus outbreak.
Industry and Occupation
Business owners were asked to answer questions as they related to the year 2018. All comparison data for industries and occupations of employment are from the 2018 County Business Patterns and Bureau of Labor Statistics Nonemployer Statistics.
Fiscal Impacts
It was not possible to collect direct information on taxes paid, value of Medicaid and public assistance, etc., as it would have been too invasive. Therefore, inferences were made about taxes paid by household income category. These effects are computed using the 2017-2019 Social and Economic Supplement of the Current Population Survey (in constant dollars) and applying the results to the survey data. Estimates for state sales tax are formulated utilizing data from Wiehe et al (2018).[11] The survey data and Current Population Survey data were treated in a consistent manner to enable comparison.
[1] In other contexts the term “weighting” is usually used to describe this procedure.
[2] Edward B. Roberts, Fiona Murray and J. Daniel Kim, “Entrepreneurship and Innovation at MIT Continuing Global Growth and Impact,” Massachusetts Institute of Technology Sloan School of Management, 2015.
[3] Some majors, such as economics, have been housed within different colleges and some of the college names have changed over the years. The categorization here follows the current UMaine college structure.
[4] Our calculations suggest that about 21,000 UMaine alumni since 1940 are now deceased (the majority of which graduated before 1970).
[5] “United States Life Tables, 2017” National Vital Statistics Reports, vol. 68, no. 7, 2019.
[6] Comparing graduate degrees to graduate enrollment at UMaine over the past 35 years indicates that the average graduate degree takes about 4.0 years. Moreover, this ratio has been steady over this period, and the less precise data from 1940 to 1985 suggest the same ratio.
[7] Lleras-Muney (2005), Meara, Richards, and Cutler (2008), Hummer and Lariscy (2011), Eide and Showalter (2011), Everett, Rehkopf, and Rogers (2013), and Clark and Royer (2013).
[8] This is unlikely to make a substantial difference to the results because the Other college category is not large. If this group is as underrepresented in the UMAA database as the average of the six colleges, then the estimate of the number of living UMaine alumni would increase by about 750.
[9] Steven Ruggles, Sarah Flood, Ronald Goeken, Josiah Grover, Erin Meyer, Jose Pacas, and Matthew Sobek. IPUMS USA: Version 9.0 [dataset]. Minneapolis, MN: IPUMS, 2019. https://doi.org/10.18128/D010.V9.0.
[10] The sample is restricted to those at least 22 years old to be more strictly comparable to the Survey data where all but two respondents are at least 22 (both are age 21, but only one reported their household income).
[11] “Who Pays? A Distributional Analysis of the Tax System in all 50 States” 6th ed., The Institute on Taxation and Economic Policy, 2018.
< Back to Alumni Impact Study Home
Study conducted by
The Margaret Chase Smith Policy Center
Underwritten by
Additional support provided by
