Towards Holistic Program Impact Measurement In Rural Mexico

The need for nuance in evaluating the impact of long-term, education-centric development programs such as PSYDEH’s

Impact measurement, evaluation, and reporting are vital yet complex and time-consuming components of any non-profit organization’s (NPO) work. Funders rightly ask organizations to demonstrate positive impact in order to justify continued support; yet, this puts pressure on NPOs to present themselves as favorably as possible, potentially obfuscating or even overstating their impact. 

Providing donors with a return on investment is important to us. Still, this does not make our impact reporting any less challenging. Given the very nature of our work, which focuses on long-term, paradigm-shifting, human-centric change, measuring progress is far more complex than simply cataloging outcome metrics in our donor reports. This article illustrates several of these complexities we face when implementing our programs in dynamic, rural environments. It also highlights why – in spite of our challenges – our work remains much needed in the communities we partner with. 

________________________________________________________________________________________________________

The days are long and the work is hard for the PSYDEH field team members engaging our partner communities in rural Hidalgo, central Mexico. Six months into PSYDEH’s 3-year human-rights-oriented, Sierra Madre Network program, the field team continues its daily work using digital tools to organize four cooperatives, with the aim of empowering women entrepreneurs to lead local projects and market their artisanal goods at a fair and dignified price. To achieve this, our team organizes biweekly capacity-building workshops for cooperative members, visits isolated communities to recruit new participants, and meets weekly to reflect on progress and chart the next steps forward. 

The apparent simplicity of these tasks belies a much more complicated reality on the ground. A typical workshop, such as a recent 10:00 am workshop held in Acaxochitlán (one of the four majority-Indigenous municipalities in which we work), could require team members to rise as early as 6:00 am to catch 3 colectivos to arrive on time – not at all uncommon, given that communities in rural Hidalgo are far apart and lack well-developed transport networks. Often returning home at 7:00 pm, team members have little time to rest and recuperate before diving into work calls or tasks that can last well into the night. This can be our reality for up to six days a week. 

In addition, team members must adapt workshop curricula to meet the unique needs of each woman and community, varying based on factors such as participants’ ages, Indigenous languages spoken, and literacy levels. For example, Jazmín Manrique Vigueras, our field leader in the Tenango de Doria municipality, frequently translates presentations from Spanish into Otomí, an Indigenous language widely spoken in the area. In some cases, field members have to modify written material into images and videos to cater to participants who average a third-grade education with many being unable to read or write in Spanish. Adapting to local contexts is an important part of our strategy, but requires deliberate effort and thoughtfulness,  prolonging the time needed to reach program goals. 

Program implementation can thus be challenging for our staff, but the same holds true for our women partners. The hours spent traveling is an opportunity cost that some women are just not willing to bear – besides weaving and attending workshops, most have to tend to the household as well. Furthermore, many of these communities lack reliable access to electricity, cell phone, and internet access, making it difficult to maintain contact with PSYDEH and their fellow cooperative members. These are some of the reasons identified by our field team for why, despite early program successes in attracting up to 75 unique women, consistent participation can be difficult to maintain. And without steady participation, the progress each cooperative can make is limited. 

Yet, whilst these challenges may get in the way of achieving certain time-bound program goals, their very existence speaks to the necessity of the Sierra Madre Network program. These communities are underserved by mainstream public services and fall outside the bounds of public attention. It is important to reach out to and empower these women precisely because it is hard to reach them. Our task is difficult, but that is an inherent aspect of our community-led development work. Far from being discouraging, these challenges reaffirm the importance of PSYDEH’s ongoing mission – to confront inequality by investing in the ground-up empowerment of rural women who often find it difficult to access the education they need to sustainably support their own development.

Crucially, these are goals that are not so readily measured by conventional metrics. Although output metrics like the amount of money earned per woman per month, or the total number of workshop participants give us useful insights on short-term impact, they do not capture the full nuance of the long-term, sustainable human development we hope to achieve. For instance, output metrics do not illustrate the full extent of the deep and generative relationships PSYDEH has built with these communities over the 15 years we’ve been working in the region. We consider these women our active partners and friends – not passive beneficiaries – who are wholly unique individuals with distinct hopes, concerns, backgrounds, and beliefs, and whose consistent participation in programming often follows only after they feel seen, heard, and valued over months and years. 

This understanding underpins our team’s approach to popular education at both a community and individual level. When reaching out to new communities, our team conducts preliminary “diagnosis meetings” to establish the context and foster personal connections. Individually, we know each cooperative member by name and take the time to learn their personal stories, their relationships with their husbands, and their personal motivations for joining the cooperatives. It is exactly this sense of being heard and cared for, of being invested in, that has compelled many local women to join our programs, the Sierra Madre Network or otherwise. In fact, many women partners who joined the Sierra Madre Network at its inception were participants from previous programs. This intimate rapport we have with the region’s women is not something easily reflected on a spreadsheet. 

This is precisely why PSYDEH goes above and beyond when reporting to funders. Instead of simply providing obligatory output metric-oriented impact measurement assessments, we also encourage them to engage directly with our programs and to see for themselves the change they are creating. We invite external PSYDEH stakeholders to visit Hidalgo and catch our team in action. We have long-term collaborations with organizations and are transparent about our progress – for example, we have done over 20 update reports, such as this, for our longest-standing global partner, GlobalGiving, the world’s leading crowdfunding platform for nonprofits. In being active participants, our funders and partners not only gain a more holistic view of program success beyond conventional metrics but also understand that it takes time to see PSYDEH’s women partners drive their own truly sustainable, resilient, and human-centric change.

________________________________________________________________________________________________________

At the end of the day, what PSYDEH promises is to live by our values of “Communication”, “Responsibility”, and “Honesty”. We have set out clear and measurable goals using traditional logical frameworks to provide an accurate picture of our impact, and are on track to achieve them. However, part of being an organization of integrity is to accept that we operate in a complex world where many factors lie outside our control. Our duty to think “big” in the kinds of impact we want to co-create with local women must be matched by an equally important duty to have frank discussions when we fall short of targets by pre-determined deadlines, and then communicate what we plan to achieve based on new realities on the ground. 

Our impact measurement and reporting journey has not been smooth, nor will it ever be. There is a reason why PSYDEH’s work is not often pursued – sustainable impact in long-term, human-centered developmental work is neither quickly achieved nor easily measured, but is necessary nonetheless. We hope our backers understand this and continue trusting us to move forward in the best interests of those partners we serve.

Author: Victoria Lim

Development Associate, PSYDEH