5Component 5How can we track the impact that our videos are having on farmers?

Component Goals

By the time you have finished this component you will:

  • Have determined what your indicators will be and how you will collect information.
  • Know how to incorporate farmer feedback to improve your approach.

Once videos have been produced and shared, it is important to learn how they are being used and what, if any, impact they may be having. This component highlights various ways that you can track video usage and measure impact. In addition, it includes suggestions for how to capture farmer feedback to better inform the creation of new content development.

The question of how to measure the impact that your videos are having on farmers is bound to be an important one. Not only will you want to know this information yourself so that you can determine which approach is having the greatest impact, but you also will likely have to report this information to donors. Some of the information you track may be pre-determined according to established donor indicators, but more likely than not, you will need to decide upon the most appropriate indicators of success with your project team. Choosing the right indicators is crucial to ensuring that you are appropriately capturing the information you are interested in for assessing impact.

This component assumes that your available resources and staff capacity for conducting ongoing monitoring and evaluation are modest. It will not cover more resource-heavy approaches, such as conducting randomized controlled trials (RCT) to measure impact. While such approaches can certainly be very robust, if you are planning to use an RCT or other intensive approach, you will likely already be working with an expert—or team of experts—who can guide you through the process.

Back to Top

What information should be collected?

To determine what data you should be collecting, the first thing that you need to ask yourself is what you want to achieve. You should have already begun this process with your Implementation Plan. Together with your team, ask yourselves what success will look like. Work together to identify indicators that are directly related to your desired consequences and that, when achieved, will illustrate that your desired consequences have been met.

One of the indicators that you are likely to capture is the number of farmers exposed to your videos. While this information can be useful in gauging farmer interest over time and comparing turnouts in different locations you are working in, it is a poor indicator of impact. Simply showing up to watch a video does not mean that the video has had any effect on the farmer's life or agricultural practices. Try to look for indicators that show some level of change has occurred. These could be indicators related to improved farmer income, changes in farming techniques, or increased knowledge of specific subjects, to name just a few examples.

You should also determine what success will look like for each of those indicators. If, for example, one of your indicators is a change in farming techniques, how will you define successful achievement of that indicator? You may consider any farmer who has adopted the new technique for more than one planting cycle as a success, or you may count anyone who has experimented with the technique. Based on how the indicator is defined, you should also set a target for your activity, such as 25% of farmers who watch a video end up adopting the technique that is showcased.

If you are able to compare this data to farmers who have not been exposed to your videos, it will likely be even more helpful for illustrating impact. A prime example of this is change in income, which may vary due to positive or negative economic factors unrelated to your project. If the average farmer who watches your videos sees their income increase by 25% over a set period, but the average change of similar farmers not participating is 30%, the videos might not be a significant factor affecting income. On the other hand, if non-participating farmer income only increases by 5%, it may mean that your videos are having a positive impact.

As part of the process of developing your indicators, you may find it helpful to create your own theory of change, which explicitly states your assumptions about how your work will lead to the changes you seek to produce. A theory of change will also help you to map out exactly what intermediary accomplishments you will need to meet to achieve your overall goals. One exercise that you may find useful is to create a chain linking your strategy to your objective via necessary outcomes.

For example, if your strategy is to use low-cost videos to share agronomic information with farmers, and your objective is to increase the productivity of farmers, your theory of change might look something like this:

Image 1

Once you have mapped out your theory of change, your team can use it as a starting point to discuss which indicators they think should be measured as part of your video activity. Write each indicator on flipchart paper. You may also want to write down why each indicator you have identified is important. Once you have finished this exercise, discuss each indicator as a group and decide how to label each one using one of the following tags:

  • Essential – This indicator needs to be measured.
  • Important – This is an important indicator, but not essential.
  • Not important – This indicator might be interesting, but it is not important to your donor, your project team, or your beneficiaries.
  • Not possible – This may include an indicator that is essential or important, but that you do not have the resources or capacity to actually measure.

You may also find it helpful to use the Indicator Selection Worksheet at the end of this component to help you to organize your team's decisions. The worksheet also includes sections for 'How?', 'Who?' and 'What?' so that you can begin to think about how these indicators will be measured, who will measure them, and what your targets will be. An example of what a partially completed version of this worksheet might look like is included below.

Indicator Relevance* How will it be measured? Who will be responsible? What are our targets?
Increased farmer income Essential Annual household surveys with farmers Government agents Farmers see 25% income growth in one year
Adoption of improved practices Essential Monthly surveys of farmers Field officers responsible for dissemination At least 30% of farmers adopt at least one practice
Increased income compared to farmers not exposed to videos Important, but not possible due to capacity limitations N/A N/A N/A

* Essential, Important, Not important, Not possible

This process will hopefully help you to determine not only which indicators you would like to measure, but also which ones are possible to measure. In the example above, although the project felt that having comparative data on income changes of exposed and non-exposed farmers was important, it was not possible given the project's limited capacity. Beware of collecting data that is interesting, but that you ultimately will not end up using. Data collection and processing can be an added burden on your staff, so make sure that you only collect information that will be useful for you.

Another thing that you may want to consider capturing is farmer feedback about your videos and your dissemination methods. This information is not directly related to impact, but can help you to improve your videos and dissemination approach. Questions may include the following:

  • Which videos that you have seen do you think were the most effective?
  • Which videos did you like least/think were the least effective?
  • How can we improve our dissemination approach?
  • How can we improve our videos?
  • What topics are you interested in that you have not yet seen?
  • If you have missed screenings, what are your primary reasons for not attending?

Periodically capturing this information will help you to continuously improve your content and dissemination methods. It will also give participating farmers a stake in your video activity. Rather than being potentially seen as an activity that is being done to them, it will show farmers that your activity is being done for them and with their input. Of course, this does not mean that you need to accept all of the suggestions you are given, particularly when they do not fall within your scope of work or current capacity. Those suggestions that you are able to implement, however, are certainly worth integrating into your activity.

It is important not to assume that one community's feedback is representative of your entire audience. Implementing a single community's suggestions without first checking to see that they have broad appeal across your entire audience may reduce participation by farmers from other communities. If you are unsure of the appeal of a given suggestion, consider raising it for discussion with a larger number of groups you are working with. You can also pilot test the changes in a couple of screening groups to see how people react. This is worth considering even when an idea does have broad appeal, as sometimes ideas that seem good on paper do not meet expectations when they are implemented.

Back to Top

How should this information be collected?

Activity Indicators

Once you have identified your activity indicators, you will need to consider exactly how you plan to collect your data. Much of this will depend upon the context in which you are working and what information you want to collect.

If you will be facilitating your videos, you can use your screening as an opportunity to collect information from your participants. Digital Green is a good example of an organization that has integrated its data collection into its video dissemination process. At each screening, the facilitator (or mediator, in Digital Green parlance) collects information on the video screened, the location of the screening, the farmers present, whether they expressed an interest in the technique or process showcased, any questions they asked during the screening, and whether they have adopted any techniques or processes showcased in prior videos. A modified version of this form, which they refer to as a Dissemination Record, can be found at the end of this component.

If you are going to screen your videos with the same groups of farmers each time, you might want to consider collecting some baseline information on each farmer when they register to participate, such as:

  • Name
  • Age
  • Sex
  • Household income
  • Amount of land farmed
  • Crops planted
  • Total yield
  • Where crops are sold
  • Specific information on techniques used that you are interested in

You can then re-administer this survey to participating farmers at regular intervals, such as every six months, to track whether there have been any changes over time. Together with information you collect on which videos farmers watch and which techniques they have expressed interest in adopting or experimenting with, you will be able to use this baseline comparative data to measure the impact of your videos on each participant.

It is important to try to incorporate periodic, randomized checks to confirm the validity of the data you are collecting to the extent that it is possible. This is necessary because individuals may sometimes inaccurately self-report. Out of respect to the facilitator, a farmer may say that they are interested in a technique or that they have already begun adopting it, even though they have not. Alternatively, they may have honestly tried to adopt the technique but have adopted it only partially or incorrectly. There is also the possibility that, for whatever reason, the facilitator has recorded the information incorrectly. Not only will random checks help to identify and correct erroneous data, but they will also help you to identify any challenges that farmers may be having with adopting specific techniques or processes.

How you collect data depends on your staff capacity and availability of resources. The most common approach is to use pen and paper to collect the information. This can be preferable if you do not have the technical capacity or resources to use digital collection methods. If the cost of using a digital method is off putting, consider first assessing what the true cost of paper collection is compared to other methods. Anything collected on paper will eventually need to tabulated, most likely by a staff person who will need to input the information into a database. The cost in terms of staff time and delays in data input can add up over time to more than what a digital method would cost over the same time period, despite the slightly higher initial investment.

Some of the digital options that may be worth considering are:

  • Mobile Phones – The simplest way to use mobile phones to collect data is by SMS using a client application hosted on a server. RapidSMS is one example of a service that uses SMS for data collection. You can use a Java-enabled feature phone or smart phone to collect data even when you are out of network range. Services such as iFormBuilder, EpiSurveyor, doForms, and others offer easy-to-create forms that can be used on mobile phones for free or at reasonable prices. You will want to check with several providers before making a decision to ensure that their services are compatible with your needs and that their prices are within your budget. Some services even include features for recording signatures or participant photos if that information is of interest.1
  • Tablets – Similar to the options available for mobile phones, many of the above providers and others offer forms that are compatible with tablets.
  • Laptops – If your facilitators already have laptops and if they are not too burdensome to bring with them in addition to other dissemination devices, they could enter data directly into their laptop using offline surveys or a simple Excel spreadsheet.
  • Digital Pens (or Smart Pens) – Digital pens are devices that convert handwriting into digital data, which can then be uploaded onto a computer. These can be especially useful if you want to also have original hardcopies of your surveys or data-collection tools. Note that some of these devices require special types of paper to function properly; this can be expensive. Unless you are already using digital pens for other purposes, however, they are probably not worth the investment given the other options available.

A great resource for learning about the latest developments in mobile tools for development is MobileActive. Their mDirectory is loaded with case studies, information on mobile tools, research, and how-to guides. Digital Green has also developed a fairly robust open-source database called COCO, which stands for Connect Online | Connect Offline. Since it was specifically built for collecting data related to video dissemination, COCO may be appropriate to your needs as well. More information on COCO, including an evaluation trial and its web-based analytics dashboard, can be found on their website. If you are interested in using COCO for your own data collection, contact Digital Green directly to discuss how it might be adapted to suit your needs.

The process of data collection becomes much more difficult if you are using a non-facilitated dissemination method. If you know the specific groups or farmers that are viewing your videos, you could mail them the baseline survey and copies of your video screening tracking form to complete on their own. Alternatively, if you are able to send a staff person to each group to facilitate the collection of data on occasion, you might want to consider that as an option. If the mobile phone penetration of the groups you are working with is high enough, you could send requests for information via SMS to each farmer.

If you are uncertain who your viewers are, it becomes a bit more challenging to collect data on them. This will likely be most common if you are using broadcast television or providing your videos on VCD/DVD to people in the community without registering them. One of the ways that you can try to gather data is by including contact information with each of your videos, along with an incentive to encourage people to contact you.

For instance, you might ask viewers to call you to receive a free seed packet, mobile phone credit, or other small gift. Individuals who call in will first need to complete a survey with several questions that you are interested in, including their contact information, before receiving the prize. You can do this by having your staff take the calls, outsourcing to a private call center, or setting up an interactive voice response (IVR) system to guide callers through the survey. Once you have people registered, you can place follow-up calls to assess any change, although given the high churn rate of SIM cards in many countries, your respondents may not have the same phone number by the time you follow up. Another idea is to partner with retailers or cooperatives to distribute the incentives and collect the information from farmers. The downside, however, to any incentive scheme is that farmers may complete the survey just to receive the incentive, which would corrupt the validity of your data. One way to prevent this is to include a couple of questions related to the content of each video in the surveys, although this can become burdensome as the number of videos you have created grows.

Try to think creatively about how you might be able to reach these unknown viewers. Above are just a couple of ideas, but there may be other more locally appropriate ways to identify and follow up with your viewing audience.

However you decide to collect data from your target audience, make sure you have clearly defined your approach in your implementation plan before you begin any video dissemination. Doing so will ensure that your team is well prepared to customize your data collection forms and that processes are in place to analyze the data that you collect. If you wait until after you have started disseminating videos or if you continually revise your plan or questions after dissemination has begun, you may end up with inconsistent data that will be more difficult to analyze for meaning.

Farmer Feedback

Collecting farmer feedback on your dissemination approach, topics and video quality can be done in a number of ways. Perhaps the most convenient, at least logistically, is to gather this feedback during video dissemination sessions. When collecting any feedback from farmers, it is crucial that you explain to participants exactly why you are collecting their feedback and how you plan to use it. You should also let them know that you encourage their open and honest opinions, and that nothing they say will affect their participation in the screenings.

If your farmers have a basic level of literacy, you can solicit feedback using a short paper survey. This should only be done if you are certain that all of your participants have a level of literacy necessary to complete the survey so you do not exclude anyone. If not, you will want to consider soliciting verbal feedback from participants. This can be done by setting aside time at the end of screening sessions on a regular basis. If you are concerned that participants may not be fully forthcoming with their true opinions in front of their regular facilitator, you may consider asking another member of your team to facilitate this discussion instead. A Focus Group Discussion Guide has been included at the end of this component. This will help to give you some ideas on how to facilitate this discussion, and will be particularly helpful if you decide to conduct a more-structured focus group.

If you do not know exactly who your audience is, collecting feedback will be similarly challenging as it is collecting data against your indicators. You will likely find it easiest to use the same methods for collecting both sets of information.

Building in a farmer feedback loop will enable you to continuously improve your content and dissemination. A simple way of doing this is to ask farmers to rate videos that they watch on a one-to-five scale after each dissemination. Using this data, you will be able to track which videos farmers prefer most, and can even disaggregate the data by gender, location, or other factors. This may be helpful in determining which videos to show to similar audiences in the future, what styles or topics are most popular, and what videos you may need to consider changing. This type of basic rating system can be easily collected during facilitated screenings. You can also use a basic SMS polling application to collect ratings from individuals who have viewed your videos at other locations by including directions at the end of the video.

Illustration of Feedback Loop

Whether collecting simple video ratings or more substantive feedback from farmers through surveys or focus groups, the most important thing is that you use that feedback to improve. You might want to consider holding a quarterly meeting with your staff to present the results of the farmer feedback you have collected. This is a great opportunity for everyone to see what is working, what needs to be improved, and to discuss what specific actions can be taken by your team. When you do make adjustments, share your decisions with the farmers you work with so that they see that you take their feedback seriously.

You should also track the changes that you make over time so that you can compare how the changes you make rate against each other. For instance, you may have decided to increase your use of narrated videos after your facilitated videos received low ratings only to find that the new videos rated even poorer. This might be a sign that you may need to ask additional questions to determine if there is anything specific about your videos that is not resonating with farmers. It may be that the issue is not related to the story style but to some other factor instead.

By continuously collecting and integrating feedback into your work with video, you will be able to improve the relevance that your videos have for the farmers you are working with. Over time you will likely find that these farmers will become more eager to participate and share what they learn with their friends and family. Ideally, this will lead to positive changes in the indicators that you are tracking.

1 For more information on using mobile phones for data collection, read Melissa Loudon's article on Mobile Phones for Data Collection posted online at: http://mobileactive.org/howtos/mobile-phones-data-collection. Although it was written in 2009, it provides a good overview of the different mobile data collection solutions available and how they function.

Back to Top

Success

Critical Success Factors

  • Establish indicators that measure outcomes and impact, not just outputs.
  • Create a feedback loop to integrate farmer feedback and make improvements.
  • Collect data using ICT tools where appropriate.

Worksheets

Back to Top

 

HTML Comment Box is loading comments...