When it comes to measuring jobs, the need to refine traditional tools with new methods

The World Bank
Alvaro Gonzalez

One of the challenges of measuring jobs is addressing impacts that go beyond direct jobs. A tool called tracer surveys is helping do this.
Photo: Chhor Sokunthea / World Bank

In addition to correctly measuring the jobs directly generated from interventions and investments, development agencies also need to estimate the resulting indirect impacts and general equilibrium effects. These are hard to measure. My recent blog highlighted the progress that donors, international financial institutions, and other multilateral agencies are making in developing standardized tools to measure these impacts. In addition to standardization, the focus is now on strengthening existing measurement tools and addressing the challenges that are left.

One of the challenges is how to address impacts that go beyond direct jobs and pick up the emergence of indirect jobs – for example, jobs that are created by new firms that emerge or expand because of an intervention. Another challenge includes measuring general equilibrium effects and considering what happens to parties that are not directly impacted by these investments, but are nonetheless affected by them via indirect channels. To do this, we are adapting a tool called tracer surveys and using techniques that are at the center of randomized control trials.

The tracer survey tool was first developed by educational institutions to track (trace) the success of their graduates. These surveys measured the performance of students before and after starting school. The difference in outcomes is attributed to the educational intervention. The same concept holds true here. For instance, we look at the number and nature of jobs in a rural village in Uganda before a road investment happens and then come back later, after the road has been built, and look at the number and nature of jobs then. Changes are attributed to the road.

The use of tracers is not new, but it can present issues. For instance, to attribute all changes to jobs created to the new rural road may not be accurate. If nearby villages where roads were not built did just as well in creating new and more jobs than the rural village that did, then all of the changes in the number and nature of jobs should not be attributed to the building of the rural road. In addition, while the rural village may have gained a number of jobs from the new rural road, surrounding villages may have lost just as many. Would the new road be judged in the same way if firms employing workers in the neighboring villages closed up shop and moved to where the new road is? For one rural village, there is a net gain in jobs; for Uganda, it is at best a neutral outcome—no new jobs were created.

The attribution issue is addressed with conscious design to understand the counterfactual—monitoring the number and nature of jobs in a nearby Ugandan rural village that did not get a road, comparing it to the one that did, and comparing both before and after outcomes. This is akin to a control group. The problem of capturing the general equilibrium effects is trickier. Again, using our example of Ugandan rural villages, it may be important to also keep track of what is happening to jobs in nearby villages that may be affected by the new road, but are not geographically close. This is trickier, but the first step is to be aware of the issue and design the tracer survey to address this. We are working on it, but there is not a solution or survey that fits all cases.

The good news is that we are tweaking traditional tools, such as control groups, with new methods. As a result, we are doing much better at addressing job impacts in a more comprehensive and rigorous manner, building confidence to stand behind the numbers.

We need to start thinking about Big Data as a way to collect data, make comparisons and do analysis and Big Data will likely be front and center of Let’s Work 2.0.