5Component 5How can we track the impact that our radio programming is having on farmers?

Component Goals

By the time you have finished this component you will:

  • Have determined what your indicators will be and how you will collect information.
  • Know how to incorporate farmer feedback to improve your approach.

Once your radio segments have been produced and broadcast, it is important to learn what, if any, impact they may be having. This component highlights various ways that you can track listenership and measure impact. It also includes suggestions for how to capture farmer feedback to better inform the creation of new content and improvements to your approach.

Measuring the impact that your radio programming is having on farmers can be challenging, because oftentimes you may not know who is actually hearing what is being broadcast. That said, there are ways that you can track this information. Most of the indicators you track to measure the impact of your work with radio you will likely already be using for your broader work with farmers. In addition to those indicators, however, you may also want to identify other indicators that you can use to determine whether radio is really achieving the impact you had anticipated relative to other ICT or traditional options. This will help you to assess not only whether radio is achieving any impact, but also if that impact is worth the investment relative to other potential options.

This component assumes that you have sufficient resources and staff capacity for conducting ongoing monitoring and evaluation. If your project has already begun, your team will have likely already developed a monitoring and evaluation plan. The purpose of this component is not to supersede that plan or the prior work of your team. Rather, it intends to provide some perspective on elements that you might want to consider for effectively monitoring and evaluating your work with radio. You may choose to add some of these elements to your overall project monitoring and evaluation plan, or simply use these indicators for internal purposes. This component is not intended to be a guide on how to conduct monitoring and evaluation activities writ large.

One of the best pathways to achieving your objectives—especially with any type of communication effort—is developing content that appeals to the needs and tastes of your target audience. This component will also look at ways that you can use farmer feedback to improve your messaging and approach.

Back to Top

What information should be collected?

Generally speaking, the indicators that you will be collecting will fall into four categories: reach, recall, adoption, and circumstance. Each of these indicators will give perspective on what impact your interactive radio activity is having. Although there is an interrelation between each of these, they are able to tell us very different things about impact.

  • Reach is an output, and tells you the number of people exposed to your activity. Sample indicators include: number of farmers who have heard your program and number of farmers who have interacted with your radio station partners.
  • Recall is an outcome, and tells you if any changes in knowledge have occurred. Sample indicators include: percentage change in knowledge on pre- and post-tests and number of farmers who can identify the key points of a program one week later.
  • Adoption is an outcome, and tells you if any changes in behavior have occurred. Sample indicators include: number of farmers who have experimented with adopting a practice and percentage of farmers who have successfully adopted a practice.
  • Circumstance is an outcome, and tells you about any changes in the circumstances of your target audience. Sample indicators include: percentage increase in income and reported change in state of well-being.

Theoretically, a larger reach will potentially lead to a higher incidence of recall, which in turn will lead to increased adoption, which would then result in improved livelihoods and lasting impact. Of course, it is not that simple. A large reach and an ineffective message could lead to a lower incidence of recall than a targeted campaign with a really effective message. In the same vein, high recall of information does not always lead to higher adoption of practice. And of course, adoption of a new practice does not always lead to improved circumstances, if it is done incorrectly or if the new practice is not appropriate for that farmer. By monitoring your activity's imprint on each of these areas, however, it will be easier for you to recognize what is working and what is not so that you can make adjustments as necessary.

You may have already considered indicators related to one or more of these categories when developing your draft Implementation Plan. As you are reading through this component, think about ways that you can improve what you have already written in your draft plan to make it more specific and relevant to assessing the impact of your work with radio. Remember to keep your end result in mind when deciding upon the indicators you will use. All of your indicators should be helping you to measure whether you have been able to achieve the change you would like to see as a result of your work.

As part of the process of developing your indicators, you may find it helpful to create your own theory of change, which explicitly states your assumptions about how your work will lead to the changes you seek to produce. A theory of change will also help you to map out exactly what intermediary accomplishments you will need to meet to achieve your overall goals. One exercise that you may find useful is to create a chain linking your strategy to your objective via necessary outcomes.

For example, if your strategy is to use interactive radio to share best practices with farmers, and your objective is to increase the income levels of farmers, your theory of change might look something like this:

Think about each indicator that you think you will need to measure to assess the impact of your interactive radio activity. You may find it useful to conduct this activity with your project team, writing up each of the indicators you identify on flip chart paper or a whiteboard. At this point, do not discount any of the indicators that are identified. Once you have identified all of the possible indicators, you can then discuss as a group which ones are most relevant to your needs. Consider dividing up your indicators into the following classifications:

  • Essential – This indicator needs to be measured.
  • Important – This is an important indicator, but not essential.
  • Not important – This indicator might be interesting, but it is not important to your donor, your project team, or your beneficiaries.
  • Not possible – This may include an indicator that is essential or important, but that you do not have the resources or capacity to actually measure.

At the end of this component, you will find an Indicator Selection Worksheet that you can use to help organize your project team's decisions. In addition to the type of indicator, the worksheet also includes sections related to how each one will be measured, who will be responsible for that measurement, and what your targets will be. Below is an excerpt of an example of a completed version of this worksheet.

Indicator Relevance* How will it be measured? Who will be responsible? What are our targets?
Engagement with radio stations Important Number of interactions made with station each month Radio station managers At least 25% increase in interactions per month
Knowledge of best practices Important Monthly surveys of farmers Field staff At least 75% of farmers recall best practices one month after broadcast
Number of farmers listening to every broadcast Not important N/A N/A N/A
Adoption of improved practices Essential Monthly surveys of farmers Field staff At least 30% of farmers adopt at least one practice
Increased farmer income Essential Annual household surveys with farmers Field staff Farmers see 25% improvement in income in one year
Increased income relative to non-listeners Important, but not possible due to capacity limitations N/A N/A N/A

* Essential, Important, Not important, Not possible

In this example, the project identified a number of different potential indicators. Some of them were considered essential, and would definitely be collected. Others were important, but not necessarily essential. In this case, the indicators deemed important would likely still be collected by the project staff if resources permit. There was also an indicator that the project would like to collect, but does not currently have the capacity for, and an indicator that they did not feel was important towards helping them measure impact relative to their desired objectives.

It can be tempting to want to collect data on lots of different types of indicators. While you certainly do want to make sure to be collecting data on indicators that are essential and, in most cases, important to measuring impact, be cautious about tracking indicators that are not related to your objectives. Data collection, input, and analysis can be time consuming. Try to make sure that you are not having staff spend their time collecting unnecessary data that you will never end up using.

Check out Farm Radio International's The Effect of Participatory Radio Campaigns on Agricultural Improvement Uptake: A Preliminary Case Study Assessment for an example of an assessment study on radio impact online.

If you plan on using more than one type of intervention to achieve your objectives, you should also consider doing a comparative analysis of the data you collect. This does not necessarily need to be done formally as a randomized controlled trial if you have limited resources. Even with more modest resources, you can at least compare data points on the same indicators for each type of intervention. For instance, if you are using face-to-face training in some communities, video in others, and radio in others, you can look at how farmers who are engaged by each method compare in terms of your indicators. The lack of randomization and control would disqualify your assessment from any peer reviewed journals, but it could provide you with helpful comparison in terms of general impact per method relative to cost.

Back to Top

How should this information be collected?

Since many of your listeners will be tuning in on their own, it can be difficult to know exactly who is listening to your programs and what impact it may be having on them. One of the additional benefits of interactive radio is that it enables you to communicate with at least a portion of your listeners. This makes collecting information somewhat easier than it is with listeners of one-way radio communication. It is important to recognize though, that listeners who interact with you do not necessarily constitute a representative sample of your audience. Therefore, you will need to consider a variety of ways to collect data to measure your indicators. For the purposes of this toolkit, we have divided up this section based on the type of listener: those who are in structured listening groups and independent listeners.

Listening groups

If you are planning to work with listening groups, they can be a great way to collect information on listening habits, adoption, and knowledge. Work together with the group leader to register their members so that you have baseline information on them before you begin implementing your activity. If you are working with radio agents, you may want to ask them to also help you collect the same types of information from individuals who use their services. Below are some examples of information that you may want to consider collecting from listening group members:

  • Name
  • Age
  • Sex
  • Phone number – to help you follow up with them later
  • Household income
  • Amount of land farmed
  • Crops planted
  • Total yield
  • Where crops are sold
  • Current techniques used

You may also have additional information that you would like to track. Include those as well, but remember to try to avoid collecting information that you will never use. Once you have this baseline data, you should aim to collect the same information from group members on at least an annual basis. This will enable you to track changes in circumstance of listening group members over time. This information can either be collected by the group leader or radio agent, or by a member of your own staff if you would prefer to have more control over this data.

Each time the listening group meets, you should also encourage the group leader to collect information on attendance, interest, questions, and adoption of techniques from prior programs. At the end of this component you can find a Dissemination Record, which is an adaption of the form used by Digital Green, an organization that works with video screening groups in India, Ethiopia, and Ghana. You may choose to ask for more or less information than is included on this form, so feel free to adapt it further to meet your needs.

More likely than not, this information will be collected using a pen and paper. If that is the case, you will want to make sure that you develop a system for collecting each of these records from group leaders and radio agents on a regular basis. This could include providing them with postage to mail them to you or arranging for your staff or local partners to pick up the forms from them individually. The sooner you receive these forms, the more likely you will be able to provide timely responses to any questions that were asked during the group session. The more responsive you are able to be, the more likely it will be that farmers will see the added benefit of participating in listening groups as opposed to listening independently.

You should also set up a database, which can be as simple as a basic spreadsheet, to compile this information. The amount of time needed for data input and analysis will depend on how many groups you are working with. It is important that your staff who will be responsible for these tasks have enough time to complete them. Otherwise, your data will just end up being a stack of papers sitting on a desk somewhere without much value for analysis. That said, there are an increasing number of mobile data collection solutions that are worth exploring, which can cut down on both cost and time. If the listening group leaders have mobile phones, then platforms such as RapidSMS, iFormBuilder, Magpi, and others offer an attractive solution over paper-based collection.

Independent listeners

There is a good chance that the majority of your audience consists of independent listeners. You may have a sense of who some of these listeners are, but you will not necessarily know exactly who is listening when and to what. Although there is no perfect way to make sure that you are capturing data on every one of your listeners, there are steps that you can take to increase your data pool.

If you are already working with specific communities on other activities, you should consider administering a baseline survey to farmers in your target audience before you begin broadcasting any of your radio programs. The survey should capture as much information as you need to establish a starting point for each farmer in terms of their current knowledge, circumstances, and practices. Ideally, your questions should focus on topics that you plan to address during your radio programming. So, for example, if one of your objectives is that farmers will use improved seed storage techniques, your survey should include a question about their current storage technique and also a question related to their knowledge about the improved technique.

This process can be time consuming, especially if your target audience has low literacy levels, but it can be an effective way to measure aggregate impact over time. You can also capture individual change over time by having respondents include self-identifying information on their survey, such as their national ID number, so that you can link them to subsequent surveys. How frequently you conduct follow-up surveys will depend on your resources, the length of your project, and how frequently you would like to assess change. Keep in mind though that people's memories are not always reliable. If you want to find out what people's reactions were to a specific program, it is better to ask shortly after it airs rather than months later.

In addition to surveys, you can ask the farmers in your target group to self-report on their listening habits and any actions they take as a result of listening to your radio program. One traditional way to do this is to provide farmers you are targeting in each community with journals that they can use to record this information. This will enable you to track listening habits much more effectively than a survey, which may be administered weeks or months after a program airs. Of course, this option is highly contingent on literacy levels and may not be appropriate in all situations.

At a minimum, the radio journals that you use should provide fields for farmers to record the date, time, and program that they listened to. You may also consider adding additional sections for reflection on what they heard or on any actions they took as a result of the program. Remember though that someone from your team is eventually going to have to review each of these journals and compile the information, so try to keep them as simple as possible. Try to review journals on a monthly basis, if possible. The more time that elapses between each review, the more burdensome the task of compiling data from each one.

Another option is using SMS or an IVR system to elicit input from your listeners. One way to do this is to ask listeners during the radio program to text or phone in their response to a particular question. For instance, you may ask them if they plan on using the technique mentioned in the program that was just broadcast or about any new techniques they have tried as a result of hearing them during your program. If your radio station partner is already using SMS or IVR to interact with listeners, then you will want to coordinate this with them.

The downside to this approach is that people may often only leave their first name or no name at all, so there is really no good way of determining how representative a sample you are receiving. To mitigate this, try to collect basic information from each farmer with whom your project or partner radio stations interact. This might include the same basic information mentioned in the bullet list above under listening groups. For example, each farmer might have a unique ID based on their name, mobile number, and village. By having this information in a well-organized database, you will be able to push out targeted surveys to your listeners via SMS.

In addition, any time someone interacts with the radio station, whether by SMS or phone, you will be able to match the number they called from to an individual in your database if you have their information. If not, you can contact that caller and ask them for that information so that you can register them. Doing this will also enable you to track whether there is any correlation between the number of interactions an individual has with the radio station and changes in recall, adoption, or circumstance.

In countries that do not allow for mobile number portability, you should make sure that you are not registering duplicates each time someone changes their SIM card and therefore their mobile number. The easiest way to do this is to check for entries with the same name to determine if it is a duplicate or someone new. If it is a duplicate, you can simply update their phone number in your database.

Whichever methods you plan to use should be included in your implementation plan from the beginning. Work with whoever on your team is responsible for monitoring and evaluation to make sure that you have the capacity and resources to implement your plans. You may also need to provide training to listening group leaders, radio agents, radio station staff, and your local partners on how to accurately collect any data that they will be responsible for.

If you are working with multiple radio stations, coordinating the collection of all of this information and managing a central database of listeners can be challenging. Your partner stations will not have the same incentives to collect some of the types of information that you plan to collect for your project. Since any database of listeners will likely be housed within each station, it can make sending out surveys to listeners in a wide geographic area difficult.

One solution is to encourage all of your partners to use the same database program, which in most cases your project would need to provide. This way you can synchronize each station's local database with a central database. You can also set up individual login credentials so that not everyone has access to all of the content. For instance, stations might only have access to their data, whereas your project has global access. Doing this may require that you provide training to radio station staff so that they know how to use the system properly. Thankfully, however, there are a number of affordable and relatively easy to use solutions currently available for these purposes, including those mentioned earlier in this component.

If your partners already are using their own—and different—databases and are not interested in changing, you may need to find a software developer to create an application that pulls data from each of their individual databases—whether that be through an IVR or SMS management system—into one web-based database. More likely than not, unfortunately, you may need to coordinate with stations individually to access and contact their listener database. If they do not already have listener databases, you should work with your partner stations to develop their own. Make sure to also provide them with sufficient training so that they are able to manage this database. If they do not see any value in keeping a database, you will need to come up with an alternative solution, such as encouraging a local NGO partner to manage it for them.

Back to Top

How can we use farmer feedback to improve our potential impact?

In Component 3, we discussed ideas for soliciting feedback from farmers before you finalize your radio programs. Once your partner stations have begun to air the agricultural programs you have developed with them, you will also find it beneficial to solicit ongoing feedback from farmers to further improve both your programs and your approach to interactivity. The process of collecting feedback, analyzing it, and taking concrete actions based on it will help your team to create an experience that is more closely aligned with the needs and interests of your beneficiaries. Doing so will likely lead to greater engagement with your audience, and through greater engagement you can increase the likelihood that your interactive radio activity will have an impact.

Since your radio station partners will be taking the lead in content development, it is best to encourage them to lead the collection of farmer feedback with your guidance and technical support. Many of your partners may not have the initial staff capacity to manage the amount of feedback they receive from farmers. Your project staff, or even better, local NGO partner staff should expect to play a large role in setting up the systems for collecting, managing, analyzing and acting on feedback.

Although it can be tempting to collect feedback on an ad hoc basis, the greatest benefit from feedback will likely come through using a structured system. This is because it can often be difficult to decide what changes, if any, need to be made when feedback is received sporadically and without purpose. Through radio broadcasts, listener databases, and established relationships in the communities in which your team works, however, you should be able to solicit fairly diverse and robust feedback from your target audience.

While you are thinking about developing a system for farmer feedback, you should also work with your partner stations to consider how frequently to solicit formal feedback. In addition, keep in mind that the amount of resources and time necessary for each method varies. Some methods will be easier to conduct more regularly, while others might require much more work on your end. This is important to consider while you are planning what your feedback system will look like. You may want to use a combination of methods to help you solicit feedback both with frequency and detail.

At the end of this component you will find a Farmer Feedback Worksheet. The purpose of this worksheet is to help you to plan what type of feedback you will collect from your target audience. It includes information on your target audience, how you plan to collect feedback, what questions you will ask, who will be responsible, and the frequency with which you will solicit feedback. The following sample will provide you with an idea of how you might use this worksheet.

Target Audience: Smallholder farmers from Eastern Province
Method of Collection Questions Person(s) Responsible Frequency
In-person surveys and call-outs
  • Have you listened to [insert program name]?
  • If yes…
    • What were the main points that you learned?
    • What did you like about this program?
    • Was anything unclear? If so, what?
    • How would you improve this program?
    • What are some other topics you would like to hear about in this program?

In-person surveys: Project field staff in each district, each targeting a minimum of 25 farmers in Chipata, Chadiza, Katete and Mambwe districts

Call-outs: Project coordinator in district office will call at least 25 registered farmers in Nyimba to administer survey over the phone due to lack of staff on the ground in this district.
One week after air date
SMS and call-ins What did you think about this program? Call or SMS us to let us know. Radio announcer to ask question on the air, responses managed by project team Immediately after airing

Your partner stations may not be able to solicit formal feedback after each program they broadcast or within each of your target communities. Even if they cannot solicit feedback all the time and from everyone, they will still likely find value from whatever feedback they do receive.

Remember though, that you do not necessarily need to make changes based on all of the feedback you hear. Rather, the feedback your partner stations receive should help to inform decisions about new programming or changes to your team's approach. It is helpful to develop a method to process and take action on feedback so that everyone on your team is clear on the process. If one of your partner stations receives a lot of feedback suggesting a particular change that you are unsure about, consider helping them to pilot test the change first. Give the pilot some time—at least a few months—and then solicit feedback again from listeners in that broadcast range. If feedback is positive, you may then consider proposing that your other radio station partners consider the change as well.

All of this is not to say that you should not welcome feedback that is received outside of your explicit calls for feedback. Try to group similar types of feedback that come in and keep an eye out for anything with high incidence. If you notice a lot of feedback on the same issue coming in, pose it as a question to a broader audience to check whether it is representative of your larger listenership or just the idea of a small, but opinionated group of listeners. Helping stations to sort and make sense of all of this feedback will be very important.

The following provides you with more information on some of methods for collecting farmer feedback that your team may want to consider employing.

Call-ins / SMS

One of the easiest ways to solicit feedback from listeners is to invite them to call-in or send an SMS in response to a question posed to them on the air. This question could be embedded into a pre-produced radio program or asked by the announcer after the program. To assure that you receive a larger response rate, consider turning the call for feedback into a raffle. For instance, you might say that one random caller will receive a free bag of organic fertilizer or some other prize relevant to your audience. This will encourage even some of your more passive listeners to consider calling in to respond. If you plan to use call-ins or SMS to collect feedback, you will definitely want to encourage your partner station to employ an IVR or SMS management system. The investment is well worth the time saved from manually logging phone calls or SMS responses. The downside of this method is that you can potentially receive feedback from anyone who is listening to the broadcast. This can impact the relevance of the feedback you receive. Let's say you are targeting smallholder farmers, but the majority of listeners who respond are merchants or larger farmers. You may be led to the possible false conclusion that your target audience wants to see a certain change, when in reality it might just be something that your secondary audience is interested in seeing.

Call-outs

If you have already developed a listener database, call-outs or sending SMS polls to a specific group of listeners can ensure that you are getting feedback from only the people you are interested in hearing from. Some IVR systems will allow you to record a message, call your sample group, play the message to them, and give them an opportunity to respond either to multiple choice questions or through voice recordings. If the vast majority of your sample group is literate, you may prefer to send out polls via SMS. There are a number of options available for SMS polls. If you plan to use this option, do some research first on their functionality and cost to find the option that best suits the needs and capacity of your partner stations. For this type of feedback to be effective, your listener database needs to be fairly representative of your target audience.

In-person surveys

Chances are that your staff or local partners will occasionally be working within the communities that you are targeting. If this is the case, you may also consider conducting in-person surveys to gather feedback and share it with your radio station partners. One of the main benefits of in-person surveys is that they allow you to reach individuals in your target audience who may not listen to your radio programs. This is a great opportunity to find out why they are not listening and to learn what you can do to more effectively engage them. The traditional way of collecting this feedback is with paper and pen, although if your staff have compatible mobile phones, tablets, or laptops you should consider using digital survey tools to ease the process. A growing number of services exist that enable you to create offline forms on these devices that can link up with a cloud-based database once the user is online. As mentioned earlier, customizable off-the-shelf mobile data collection solutions already exist. Do your own research first to find the solution most appropriate to your needs. Obviously the big downside of in-person surveys is that they can be time consuming to conduct.

Focus groups

Another option for in-person feedback is to conduct focus groups with your target audience. Unlike surveys, which are pre-defined, focus groups allow for follow up questions to explore a response in greater detail. They also provide an opportunity to play a program or segment of a program, which allows you to ask for specific feedback on what was just heard. Of course, it is also easy for a focus group to be hijacked by a couple of very vocal respondents. You will need a trained moderator to ask clear questions, engage all participants, and limit the responses of more vocal respondents to within reason. A Focus Group Discussion Guide has been included at the end of this component as a reference for conducting your own focus groups. Your partner radio stations may not have the capacity to organize and facilitate their own focus groups, so your project staff or local NGO partners may need to conduct this on their behalf. If this is the case, make sure that the questions asked during the focus group are driven by what your partner stations are interested in learning.

Other options

Although much of the feedback you receive may be more qualitative in nature, you may also want to consider building in an interactive rating system for each of your programs that air. For example, at the end of each program the station might include a message such as, "Tell us what you thought of this program. Send an SMS to 39555 with the word FARM followed by a rating of 1 to 5, with 1 being terrible and 5 being amazing." This will enable you to compare which programs are most popular with your listeners. Through this process, you may notice similar elements that exist in the popular programs that do not exist in the less popular ones. If this is the case, you may want to consider re-scripting and recording a less popular program to include those elements and see what happens.

Regardless of what type of feedback your team is collecting, you should also consider scheduling standing meetings with any staff and partners involved in your interactive radio activity to review and discuss that feedback. By holding a regularly scheduled meeting—say on a quarterly basis—your team will have an opportunity to learn from and react to this feedback, enabling them to make any changes in a timely manner. If you are working with more than one radio station, consider analyzing feedback from each one to see if you can identify any common themes. This information will likely be useful for each station to learn, and also for your project team to help you better structure the support you are providing to your partners.

Whenever your team does decide to make changes based on feedback received, encourage your partner stations to advertise those changes with listeners so that they know you heard them. For example, before a new program is played, the announcer might read a message such as, "We've heard your thoughts on what type of programming you want to hear, and I think you'll find this next program to be just what you are looking for." The exact message will vary, but the important thing is that your audience knows that you value their input and are taking action based on what they tell you. This will likely give them a more positive impression of interacting with your team and hopefully lead to greater loyalty to tune in. Increased interaction and loyalty from your listeners are not sufficient to achieve your objectives on their own, but they will likely improve your odds.

Back to Top

Success

Critical Success Factors

  • Establish indicators that measure outcomes and impact, not just outputs.
  • Select collection techniques that will reach a representative sample of your target audience.
  • Build capacity of partner radio stations to manage farmer feedback.
  • Systematically solicit farmer feedback and use it to make improvements.

Worksheets

Back to Top

 

HTML Comment Box is loading comments...