| Communities most often do simple pre-post tests, using the instruments for both parents and youth that can be found in the teaching manual. This quick and easy type of evaluation can be done without extra evaluation funds. By contrast, in the scientific, longitudinal studies extensive information is collected in in-home interviews with both parents and youth, including videotaped interaction of the families taking part in structured interactions. This type of evaluation is extremely expensive and out of reach of most community-based evaluation efforts.
There are two ways of using the short surveys found in the teaching manual. The surveys can be given as a retrospective pre/post test and at the end of the last session or as true pre- and post-tests, given at the beginning of the first session. The true pre- post- may be more prone to bias than the retrospective version for the following reasons.
Retrospective Pre- Post Test Survey
In order to collect valid results, it is imperative that the retrospective surveys be administered very carefully. If participants record their answers in the wrong column the resulting data will be useless. Use the following steps to ensure valid responses:
- Before handing out the surveys, use strips of paper to cover the second column so that respondents can only see the first column.
- Read each item and ask participants to circle the appropriate response, rating their behavior NOW.
- After all items are read, have them turn back the strip of paper to reveal the second column.
- Read each item again, asking participants to rate their behavior BEFORE the program.
- When reporting findings results using the retrospective pre- post-, be sure to clearly describe that participants are rating their own degree of change
Note that the surveys with the NOW and BEFORE columns are found in the teaching manual.
True Pre- Post Test Survey
1) In the first session individuals often want to "put their best foot forward" and so they may overestimate the rate at which they do positive behaviors in order to "look good." Because of this response set, it is quite hard to show positive change with the program. For example if they rate themselves at a "3" or "4" (a good bit of the time or most of the time) before the program, there is little or no chance for positive change.
2) Another way in which the true pre- post- is prone to bias is that people understand the items differently after they have gone through the program. For example, there is an item about frequency of family meetings and before the program participants may think they get together as a family quite often, whereas after they understand what family meetings are, they are likely to rate themselves lower, even though in reality they may be conducting family meetings more than before the program.
3) The final disadvantage of doing a true pre- post-test evaluation design is that participants, who may have come reluctantly in the first place, may be turned off by being tested the minute they walk in the door.
On the other hand, some funding organizations require the true pre- post-test evaluation. If this is the case, simply use the version of the survey that has only one column of possible responses.
In reporting findings from the surveys it is suggested that the four response categories are collapsed so that “a little bit of the time” and “some of the time” become “low” and “a good bit of the time” and “most of the time” become “high.” Then report the number and percent of respondents reporting “high” after the program, compared to before the program. See the following example:
Parent Survey
n = 539 parents reporting
Item |
Number reporting
“a good bit of the time”
or
“most of the time”
Before |
Number reporting
“a good bit of the time”
or
“most of the time”
After |
Wait to deal with problems with
my child until I have cooled down |
95 (18%) |
392 (73%) |
Help my youth understand what
the family and house rules are |
208 (39%) |
471(87%) |
Psychometrics are available for retrospective pre- post-test surveys. Please contact Catherine Webb at 515.294.1426, cwebb@iastate.edu.
Survey Items by Risk Factor
Some groups wish to analyze survey data by risk factor. Following is a list of risk and protective factors followed by specific survey items:
| Youth Survey |
Parent Survey |
Positive goals for the future – 1 |
Communication style – 1, 3, 10, 12, 15 |
Relationship with parents – 3, 7, 8, 14 |
School involvement – 9, 14, 17 |
Stress management – 2, 6 |
Nurture and support – 2, 4, 8, 11, 13, 16, 19 |
Peer pressure skills – 4, 11 |
Contingent parenting – 5, 6, 7, 18 |
Family communication – 5, 10, 13 |
|
Following rules – 9, 12 |
|
|
