APPAM 2022

Recap of the Association For Public Policy Analysis & Management Fall 2022 Conference

What is APPAM?

APPAM 2022 is the annual conference for policy analysis.

Why is APPAM important?

Academic researchers are (usually) slow-paced and rigorous. Government bureaucrats are (usually) time-crunched and not read-in on the latest (social) science. At its best, APPAM allows both groups the time and space to connect, allowing the groups to translate academic findings into action by government to better serve the American public.

Who attends APPAM?

The mix skewed mostly academic, followed by think tanks (Pew, Urban). There was reasonable government representation and a handful of folks from industry (I see you, Mathematica). Nonprofits (that aren’t think tanks or funders) were very rare.

GSA’s Office of Evaluation Sciences was out in force, with a 1/2 dozen person contingent. There were Feds from Census, CRS, GAO, Federal Reserve, HHS (ACF, AHRQ, CMS, CDC), OMB, SEC, USDA and VA. State and locals were rare.

The conference was 95%+ American. Most of the foreigners were from East Asia (Korea, Singapore), with the occasional European researcher or bureaucrat.

Takeaways

Policy papers have a status hierarchy

My informal observation of the hierarchy of policy papers…

• Theory — hardest to publish, become legend if successful (read: Weimer & Vining)
• Methods — tyranny of small differences (read: but my methods ekes out marginal signal improvements under these very specific conditions)
• Quantitative — datasets are the rate-limiting step (read: existing structured datasets have mostly been mined of the juiciest publication nuggets; assembling a de novo dataset is hard, time-consuming, expensive AND won’t get you tenure)
• Literature Review — hard to publish if you’re not already a big name (read: are the profession’s Strunk & White)
• Qualitative — struggles for a fair hearing because of the technochauvanist impulses of economists (read: so, where’s your regression?)

Quantitative papers were by far the most numerous, and follow the structure…

• Connection to theory (read: chant the correct incantations)
• Pre-registered hypothesis (read: lets make it more difficult to p-hack)
• Data Set (read: lord help you if you spent years constructing a de novo data set, only to find a useful, but alas null, result)
• Results and attendant statistical methods (read: prove rigour)
• Optional: Normative takeaways (read: the most important bit for government)

Policy folks slay at taking advantage of natural experiments

Policy decisions with hard cut-offs make for “great” discontinuities [1], and arbitrary discontinuities make for great natural experiments.

Paper: A jurisdiction changed its funding method for COVID small business loans from a community block grant to local funding. The community block grant required applicants submit a bunch of paperwork; the local funding didn’t. Queue interrupted time series to show the equity impact of reduced documentation burden on under-served small businesses. Underserved business benefited (+3%), but non-underserved businesses benefited more (+10%), increasing the disparity.

Paper: GAMAM (Google, Amazon, Meta, Apple, and Microsoft) acquire more in some software markets than others. GAMAM Acquisitions reduce start-up formation of related businesses by 2/3rds despite only being 5% of acquisitions supporting the “kill zone” hypothesis of big tech. Acquisitions by private equity (14% of acquisitions) actually increase start-up formation supporting the “entry for buyout” hypothesis. 3/5s of acquired brands are discontinued.

Poster: Americans older than 65 were eligible for vaccines earlier than younger Americans, which means an arbitrary discontinuity, and there’s a method for that — regression discontinuity design (RDD). Vaccination didn’t induce risky behavior, debunking the vax’d old people flocking to night clubs myth.

Sad, but entirely unsurprising, results abound

Quantitative Poster with descriptive statistics: 1/2 of Americans that don’t trust science think it is justified to harass or threaten a public health official. The numbers segmented by Republican vs Democrat and educational attainment are lower, but break exactly the way you’d expect. For example, Republicans believing “harassing public health officials is justified” rose from 20% to 34% from November 2020 to July/Aug 2021.

Quantitative Paper with traditional regression analysis: Contract bundling hurts small and disadvantaged business to the tune of several billion dollars a year. Briefly, contract bundling is the procurement technique of combining procurement into mammoth contracts only a few large consultancies can bid on with the prima facia justification of increasing efficiency and reducing cost. Including small business subcontracting rules into the bundled procurement doesn’t help.

Quantitative Paper with simulation: Small tweaks to the distribution method of COVID funds have large equity impacts; but don’t worry, the tweaks are rarely used. First come first serve disadvantages the most ( mitigate a bit by create a fast track for underserved business, but otherwise avoid. Lottery ( use higher odds for underserved businesses). Points system ( give points for underserved demographics, but use individual owner factors NOT area-level factors ). Don’t use pre-COVID economic factors, because it perpetuates disadvantage.

Lit Review Paper: Black Americans are disadvantaged at every single step of the business formation process (impediment).

1. Interest (Macro-factors)
2. Opportunity recognition (Self-efficacy)
3. Networking / Nascent entrepreneurial behavior (Access)
4. Resource acquisition (Legitimacy)
5. Resource coordination (Scale)
6. Value capture (Ambidexterity)
7. Innovation and renewal (Reactivation)

(1) Macro: SBA’s shift towards “race neutral” policies decimated black business, 38% of SBA awards by dollar went to black businesses in 1980 vs 10% in 2006 (Nopper 2011).

(3) Networks: Even when blacks form their own (because of exclusion) or are admitted to networks, they recieve fewer benefits (Pedulla and Pager 2019).

(5) Resource acquisition: White households have 10x the wealth of Black households (Mcintosh, Moss, Nunn, shambaugh 2020). 1/5 of the difference in entrepreneurship between blacks and whites is because of this household wealth difference (Fairlie 2018).

(6) Value capture: Whites prefer white owned business (Ogbolu and Singh 2019), expect lower prices when they deign to shop with black-owned business (Younkin and Kuppuswamy 2019) AND have overt animus towards black business (Smith 2021).

Lit Review Paper on Regulations: The US is an outlier in not having public interest data laws.

Korea: 2013 Act on promotion of the provision and use of public data

France: 2016 Law for a digital republic
Includes the part of private data that needs to be used for public purposes, and obligates firms to open some private data for public interest purpose.

European Union: 2022 Data governance act
Allows individuals and companies giving their consent or permission to make data public for public purpose.

Japan: 2016 Basic act for promotion of utilization of public-private data

Randomized-controlled trial (RCT) experiments are the gold standard, but are difficult to pull off

Quantitative Paper > RCT: A/B testing emails to jurisdictions to apply for state and local recovery funds ($20B available to 26k jurisdictions). +3% by providing an action summary. +1.5% from action-oriented subject line. No change for peer comparison and reminder emails.

Qualitative methods are boss

I left with a reinforced appreciation of the importance of qualitative methods. Qualitative research methods:

1. Transform (mere) “anecdotes” into data.
2. Access features that are difficult or impossible to judge quantitatively.
3. Confirm that quantitative measures capture what you think they do.

Qualitative paper: Black Entrepreneurs are “excluded by design”

Facing pattern matching that deems them “not ideal workers”, limited network access, differential treatment and enduring exclusion, black tech entrepreneurs in the Bay Area adopt “Survivalist” strategies including emphasizing credentials, emphasizing middle class identity, performing unpaid DEI work and codeswitching.

Qualitative paper: Despite abysmal diversity, tech workers still believe their firms are “Better Than Most.” Rationalization strategies include…

Relative comparison

• “At least we’re not Uber”
• “We hire CS grads outside of Stanford, unlike my girlfriend’s company”

Evidence of effort

• “We share diversity numbers, even though we don’t have to”
• “But we offer diversity training!” ( read: performative diversity efforts )

Expansive definitions

• “I hear different languages at work”
• “We have white people from Canada, Chicago and New York City, not just Stanford” ( my personal favorite )

These rationalizations normalization unpaid/unrewarded diversity work, and justify the avoidance of methods that actually work, like affirmative action.

Footnotes

[1] “Great” means “not great.” Discontinuities are generally bad policy design. They cause market distortions by incentivizing people to change their behavior in otherwise harmful or irrational ways. I believe the harm that discontinuities cause has been established for at least 50 years, yet they remain common. Know better != Do better because “good” politics (almost always) trumps good design.

All views are my own, and do not represent those of my employer.