A wise man once said “praise publicly, criticize privately.” This week we saw a handful of blog posts and associated comments regarding the PASS Summit survey results. If you review the posts and comments I think you would find them more along the lines of “criticize publicly, remain silent privately”, because that’s how I feel since nobody bothered to do some fact checking before going public with their posts.
I’m not saying that the posts are wrong. I recognize the need for some healthy discord. I do wish, however, that we could get a few facts addressed and I would like to do that now.
Accidentally On Purpose
I have lost count of the number of times over the past weeks that the word “bias” has been mentioned with regards to the surveys that PASS has composed in the past. And I will not argue with anyone on this matter. What I do want people to ask themselves is this: Do you think the questions were constructed in a specific way in order to be biased? Because if you do, then you are giving us way too much credit. Believe me, we’re not that clever, and we are not sitting around a table trying to craft questions in such a way so that they have a bias. Any bias that exists in the survey is simply dumb luck, as it were.
Customer Feedback Is Important
Fairly obvious, right? Well, that is the whole point of the surveys that we send out. We need feedback from everyone in order to make certain we are steering the ship in the right direction. To the 200 people that answered the Summit survey I want to say “thank you”. To the 2,000 people that skipped completing the survey I want to say “I understand”. I wouldn’t want to answer so many questions either. I recognize that most people don’t want to answer more than seven questions in any survey. We will simply need to figure out a better way to get all the information we need.
We Need Help Putting Together Surveys
This one is rather obvious by now, to everyone involved. We need help in putting together the questions, the way the surveys are distributed, the way we can entice people to participate by responding, how the results are interpreted, and how we announce the results publicly. We need help in each and every area with regards to customer feedback. I am not trying to slam the work that has been done previously by PASS, I am only trying to make certain that PASS knows they need some help and that the Community knows that PASS knows they need help. Hopefully we can come together to agree on that point.
Now, I’ve seen a lot of people complaining, either about the survey itself or the fact that the PASS Board has not come out with an official announcement or whatever else they feel the need to get off their chest. What I haven’t seen offered up by anyone is a solution, only the complaints. I also never like it when a complaint is not followed up by a suggestion for improvement. So, I’ll be the adult here and offer mine:
Part Time Is Not Enough
We do not have anyone dedicated to putting together our surveys. And at the same time we recognize just how important it is for us to keep our fingers on the pulse of our Community. I would say that the time has come for PASS to either dedicate a resource to gathering community feedback, or we outsource to composition of our surveys to a professional service with experience in this arena. We cannot afford to continue in the same manner that we have been operating. The information is too valuable, we need to start treating it as such.
My solution is exactly that: We must create a position that is dedicated to polling the PASS Community in such a way that we get continuous feedback. There are a lot of options for PASS to explore here; we could do surveys, we could ask 3-5 questions of each member at various times throughout the year, we could do informal polling during the Summit, we could find a way to do some polling at SQL Saturdays, etc. But none of that will matter unless we have someone in charge, someone coordinating the efforts, aggregating the results, interpreting them, and communicating the results back to the Community along with an appropriate action plan. This person could be a current staff member (which means that we would need to shift some current priorities and assignments) or we could look to outsource our needs. Either solution would come with a cost (either hard dollars or staff time) and right now we don’t really have extra resources on hand.
So, I’ll put it to the Community: leave your thoughts below on how we can implement a low-cost solution to gather, aggregate, interpret, and report an action plan based upon our collections. I’ll take the suggestions left here to the PASS Board as well as the PASSCAB and let’s see if we can work together on a solution.
[This post is my opinion, and mine alone. It does not, in any way, represent an official statement from PASS. If you think it does, you’re wrong.]
Hey Tom.
The “currency” of social media is transparency. If bloggers/microbloggers want to maintain credibility they are expected to convey what they really think. I agree with your quote for the individual and sometimes I think it even for organizations. The problem is that when you have an organization that has a history of communicating what it was going to do and it has failed to deliver on those communicated actions – volunteers will become disheartened and less inclined to volunteer their time moving forward.
In that context, it doesn’t persuade former volunteers to reconsider the idea of volunteering for PASS when they are told that they cannot speak freely unless they first recommit to PASS volunteer-ism.
I think that your April Fool’s joke about PASS being renamed to Passé had a kernel of truth to it. If PASS doesn’t start participating in a transparent conversation with the SQL Server community that allows the community to communicate honest criticism, PASS will definitely become out-of-date.
Chuck,
I did acknowledge the need for healthy discord, so please do not think that I am telling people to not speak their minds.
However, if you are going to be critical, it never hurts to offer a solution at the same time. I don’t see any solutions being proposed, just criticisms. You cannot continue to beat someone down and think that you going to enact a positive result.
I’m not sure I was criticizing the survey. I was commenting more on the answers and Brent’s interpretation.
That being said, I think there is a level of frustration with PASS that over the years they have been criticized in private and used that privacy to continue to status quo and ignore change.
You on the board, meaning all of you, need to get past the idea that you can please everyone, and that you won’t get criticism. You cannot get defensive at comments. What you can do is step back and analyze what people are saying, and look go engage them, publicly or privately.
You are elected officials, and while I completely understand that you can’t do this full time, and I accept you will make mistakes, what I don’t accept is a lack of communication, openness, and responsiveness.
I have to say that you, Thomas, do a better job than almost anyone in communicating back with the community. I appreciate that, and I thank you for your efforts.
Aw, shucks. *group hug*
I try my best, where I can. Thanks for noticing.
Tom,
I’m sorry if the Board feels like they are being beaten down. I wouldn’t waste my time talking about PASS if I didn’t think it had no value. If I were on the Board, I’d view community feedback as a great opportunity to begin to engage with us.
Thanks for your invitation to suggest solutions.
Here you go:
I think more people would be interested in PASS if they knew they could easily afford the Summit. Many of us have no idea if we will be able to attend until we hear back from our employers or if our checking account shows us some love. If the PASS Summit was priced as attractively as possible, more people would commit to it sooner and therefore be more engaged.
To do this PASS can focus on less things, spend less money, lower costs.
PASS could reduce expenses by utilizing videoconferencing more to reduce airline, hotel, hotel conference room costs.
I’d also like to see more Board members be on twitter. I think that would go a long way to facilitate communication. This is only a preference of mine, I don’t hold anything against Board members that decide not to be on twitter.
I agree with Steve. I think you do the best job on the Board of engaging the SQL community.
Basically Tom, you guys are on the firing line. And it’s a volunteer organization. You’d get a lot less complaint when you’re paying people, not because they’re any less unenthused by what you’re doing, but because you’re paying them. It’s just the nature of the beast. Besides, not everyone posted something critical of the survey: http://scarydba.wordpress.com/2010/04/22/pass-summit-content-survey-results/
Two things – first, “praise publicly, criticize privately” works great inside companies and between friends, but not for political organizations like user groups. Think of guys like me and Steve as journalists posting op-ed pieces.
Second, hire a survey company. This is what they do. I used to work for Unifocus and I think the world of ’em, but shop around. Having database professionals design an opinion survey is like having survey professionals design a database (and believe me, I’ve seen it happen). I wouldn’t design a survey like this myself, and I used to work for a survey company. Bring in the pros. It’s cheap, and it’s worth it.
I wasn’t originally going to post this second part, but it frustrates me that I even have to say it out loud.
Every time someone criticizes PASS, they’re offering a solution. They may not be wording it the way you want to hear it, but it’s there. For example, when I listed the top ways to make sure readers ignore you (with a jab at the SQL Server Standard), I was literally giving PASS a checklist of things that needed to be fixed. When I write “Don’t offer an RSS feed or email subscriptions,” read between the lines and understand that to be successful, you should offer an RSS feed and email subscriptions for new versions of the Standard. When I say the survey questions are biased and they weren’t written by survey professionals, the answer is to get survey professionals involved.
It sucks being on the receiving end of criticism. I know. I’m there all the time. But behind every valid criticism is a fix. Only the invalid criticisms like “You suck” have no valid fix. If you’re hearing “You suck” on anybody’s blog, call it out. But if they’re pointing out valid problems…
Brent,
You suck.
I believe I acknowledged the fact that we need help with our surveys. I also believe that by now people should know that we know we need help. I think my real problem here is with the bias of the questions, as in “PASS created these questions with a purposeful bias.” To me, at least, that seems to be the general sentiment. I could be wrong on that, but it is what I have been reading between the lines you have helped to paint so far.
Ha! I hear you, but I’m not saying PASS purposely created biased questions. I’m saying that without a survey pro, it’s all too easy to accidentally do it.
For example, I’ve seen compound questions like, “Do you want to move the Summit away from Seattle even if it means less participation from Microsoft?” That’s a valid business question, but not a good survey question, and it produces biased answers. I understand why the business (PASS) needs that information and why it would be asked that way by well-intentioned people, but that doesn’t mean it’s not biased.
When I tune a bad query, I don’t think a developer of doing it maliciously. It’s just a bad query, that’s all.
Tom,
I am late to this party. As an independent observer I would recommend that PASS abandon transparency plans for the moment and go closed door ‘benevolent dictatorship’. I do believe that PASS’s intentions are good, and that they do want the best for and believe they are acting in the best interests of the community. However, there does appear to be a lack of a cohesive strategy, and watching it play out on the blogs, and in the comments of blogs, while entertaining, probably does not reflect that well on the organization right now. I would disagree with Brent on hiring the survey organization. I’d say that was a waste of money because really you had already made the decision to stay in Seattle. If you know what answer you want then there’s no point putting together a great survey :)(PS – I have been guilty of this before)
And so this isn’t taken as criticism without offering a solution here’s my recommendation:
– identify your key target audience (in my opinion this is your chapter leaders)
– do everything you can to make that target audience successful (if it’s chapter leaders help grow their attendance at user groups and SQL Saturdays), provide them a platform to get the word out, negotiate good rates with different providers of services, give them top notch material to re-purpose. (train the chapter leaders on presentation techniques, how to put on a great event etc.)
– you widen the pool of PASS HQ evangelists, you get more news out and input back from different groups.
– you become no longer reliant on a single event, but make the tons and tons of events that go on around the world every month more impactful.
Just some thoughts from someone interested, but not in the DBA club.
Oooo, you know, Christian’s right about the survey thing, and I’m not just saying that because he’s my boss. (Although that IS why I’m fetching him coffee right now, but that’s another story.) It reminds me of one of the questions the survey people used to ask when they were done with each question’s design. They’d say, “Fast forward three months. The survey comes in, and the answer is overwhelmingly in favor of answer A. Now what do you do?” They’d ask that of each answer (even ratings questions) and that helped the client determine what other followup questions to ask. Sometimes, though, all of the answers would be the same – “We don’t really have an action to take here.” In that case, the question comes off the survey.
For PASS, if the BoD has overwhelming factors that require the Summit to be in the same place, then it doesn’t matter what the answer is – take that question off the table. Focus the questions on what you really need guidance with, or don’t survey at all.
Christian rocks. (Again, not because he’s my boss. He’s stuck with me regardless.)
Wow – I have Brent down in writing saying I was right about something. I’m in shock. It’s defibrilla________________________________________________
Heh heh heh…all part of my master plan to get Christian’s laptop. SCORE!
I agree with Brent, that he & Steve are writing op-ed pieces, and nothing wrong with that. It’s a useful function to have someone thinking and commenting. As long as anyone commenting is trying to make things better, I’ll listen to what they have to say.
I don’t think every complaint needs a solution. I think it’s worthwhile if you have ideas to list them clearly, and not make a probably slightly distressed reader/board member have to pull out the “fixes” – but that’s just me.
Tom, that’s a good you make about accidental bias, which is exactly why I believe that if you are going to do surveys, you should get professional survey people to help. This is an attempt to mine for business intelligence, not just decide what kind of pizza to order for the company party. Therefore, having even a Full-Time staff position dedicated to surveys is not effective if they are not well-trained in the area of survey preparation. In this case, I’d say “buy” is a better option than “build”.