Friday, January 15, 2016

Home Grown Surveys ... for the Record

Having reviewed still one more home grown survey, I would like to state, for the record, the problems I have with home grown surveys.  Undoubtedly you will conclude that I am biased, since GraceWorks has its own proprietary survey product.

Struggling right now through another rather difficult custom survey project, there are times when I wish homegrown surveys would in fact do the job - it's frankly a lot of work!

In priority order, here are my concerns:

(1) Is it a good score?  Without comparison data you tell me.   (e.g. Please rate the overall customer service you and your family receive from ______________:    Excellent 46.60%, Very Good - 32.04% Good - 14.56%, Needs Improvement - 6.80%.  Help me understand, please: Is that good or bad?)

(2) How much do parents care about this item?  In other words, even if its effectiveness is not stellar, does it matter?  This is almost a tie with (1) in concern.  For example, for a Christian school, is "Teachers exhibit care and concern for students" more important than "Use of technology in Instruction?"   Answer:  The former is more important, BY FAR.  (How do I know?  Asked over 35,000 parents ... )

(3) If a program element is done poorly or well, to what degree does it impact overall satisfaction?  The assumption of our survey is that Christian schools have limited time and money, and therefore we need to pick out the program elements to improve that REALLY MATTER.   That's quite different than program elements that happen to strike my fancy ...

(4) Scale problems.  There is a whole book, (The Ultimate Question, 2nd edition) -  not to mention numerous websites - on the correct scale to ask the willingness to refer question:   On a scale of 0 to 10, with 10 being high, how likely is it that you will refer our _______ to a friend or colleague?  This is not a copyrighted question.  How to score the question can easily be found as well.   Don't ask it Yes / No, Don't ask it in 4 or 5 point scales.  On five point scales, there are OCD / Perfectionists types who will never, ever give you a 5.  No matter how much you deserve it!  And 4 misses the cut on the top category of willingness to refer. 

It took Fred Reichheld 45 years to figure this all out - why mess with a good thing?

(5) Testing multiple aspects in one question.  In general, if your question or answer has an "etc." this is NOT GOOD.  Consider this question:  "Has your student ever been a victim of bullying, teasing, harassment, etc. while attending ________."  Can anyone answer "No" to that question without lying?  And do we think for a minute these are all equivalent?

(6) No cross-tabbing.  It's not just how many people answered a question a given way, but how does answering the question that way correlate with other important things, like overall willingness to refer?

Let's say that 50% of respondents check the box that says that coming up with the cost of tuition is a large sacrifice for them.  Is that good or bad?  Now consider if you knew that those 50% also were moderately MORE SATISFIED with the school than anyone else, and the difference was statistically significant ... How might that change your view of people sacrificing to pay your tuition? 

(BTW ... that is the typical result when we ask the question.)

(7) Opportunities lost ... to recruit volunteers, ask for leads, keep tabs on your alum.  Of course, this requires you to vary questions asked based on the answers to other questions.  Can be done, for sure, but difficult and painstaking for people who don't do it all the time.  (It's painstaking for people like me who DO do it all the time.)

OK ... I feel better now.  Back to another painstaking custom survey setup.


(c) 2016 Dan Krause GraceWorks Ministries All Rights Reserved\

No comments:

Post a Comment

Thank you for posting a comment. We enjoy interaction with our readers.