How to Measure MTW Success: Lessons Learned from Studying the Moving to Work (MTW) Program
By Dr. Keely Stater | Director of Research and Industry Intelligence | Public and Affordable Housing Research Corporation
A key part of measuring success in the Moving to Work (MTW) program as a whole is understanding how policy trade-offs and data nuances can impact overall performance results. Our new white paper ‘Local Decision-making in the MTW Program,’ investigates the policy trade-offs made by MTW agencies, building on data from recent MTW studies conducted by Abt Associates. Below are some of the takeaways we learned on how to approach measuring MTW performance outcomes, interpret MTW data, and understand the localized policy trade-offs driving program results.
- Carefully chosen comparison groups are necessary.
MTWs agencies are significantly different from the population of housing agencies in ways that impact agency performance. MTWs tend to be much larger than the typical housing agency, more frequently located in urban areas with tight housing markets, and serve communities with greater needs. As a result, comparing MTWs to all other housing agencies on program outcomes leads to inaccurate conclusions. MTW/non-MTW comparisons require a carefully selected peer group of non-MTW agencies that are similar in size, rental market, and community needs to each MTW counterpart. The best method of comparison is to create a peer group for each MTW agency by weighting data from multiple non-MTW peers with like features.
- HUD data systems do not tell the whole story.
MTW data do not ‘fit’ cleanly into HUD systems. Moreover, a large amount of MTW data do not appear in HUD systems. One estimate found that just 82% of voucher expenses appeared in the Voucher Management System (VMS) while MTWs spent 99% of their budget justification in total. Additionally, each agency may enter data into HUD’s systems differently due to the variety of activities implemented across agencies, different interpretations of data fields, or variations in the agreements held between MTW agencies and HUD. Most importantly, HUD data systems are not designed to measure MTW program outcomes and thus do not capture the many relevant data points. These factors make conclusions that rely solely on HUD data to assess program outcomes inaccurate, since they do not present the full picture of MTW activities or adjust for the data nuances found in HUD systems. Conversations with MTW agencies are essential to understand the specific definitions they use as they input their data and which critical outcome data may not be included. Additional survey data are also needed to more clearly represent MTW program activities and their intended outcomes.
- Benchmarking is important.
Even a carefully designed comparative analysis may not be enough to provide the full picture of MTW impact. It is also important to compare an MTW agency to itself over time. For example, some MTW agencies may seem similar to their peers on public housing occupancy, but have made impressive strides in reducing vacancy since joining the MTW program. These changes are an important part of the MTW story. For example, while there was no difference in the average public housing occupancy rate between MTWs and their non-MTW peers between 2000-2015, more MTW agencies had improved their public housing occupancy rate during this time period.
- Factor in opportunity investments.
MTW agencies are not only serving their clients now, but looking to the future on how to serve their communities. They are building their programs to be sustainable and forward-looking. As such, preservation and investment in savings and administrative infrastructure are important inputs into future success and should not be discounted in the present. For example, differences in average reserves between MTWs and their non-MTW peers can largely be driven by the need to leverage these dollars in development deals that bring additional units of affordable housing to their communities. Likewise investments in preservation today will stem the future loss of affordable units. These future investments also need to be considered when assessing program impact.