Clients often ask us whether early or late responders are more positive or more negative in their ratings. This not only is an interesting question (at least to us, especially when we are in nerdy moods, which is fairly often), it has important ramifications.
We analyzed a random sample of sixteen recent surveys to see how overall engagement varied by response-time increments of 10%; that is, we looked at the average engagement scores of the first 10% of responders, the next 10%, etc. In each case there was a general upward trend. Averaging the scores across all sixteen studies resulted in the following graph:
When we looked at the above graph, it seemed to us that there were some natural breaks between deciles where we could see differences. This resulted in the following graphs:
Discussion
We call people “highly engaged” if they rate the company as a 6 or a 7 on our seven-point engagement scale, and disengaged if they rate it as a 1, 2 or 3. Only 44% of employees in first 10% of respondents were highly engaged. This percentage rose to 54% for the next 40% of respondents and 59% for the last half of respondents.
Fourteen percent of the first 10% of respondents were disengaged. This dropped to 11% for the next 40% and eight percent for the last half of respondents.
What is going on here?
We believe that there is a propensity for some dissatisfied respondents to pounce when they have an opportunity to provide feedback via an employee survey. After the survey has been open for a while, the percentage of respondents who are disengaged drops.
We believe it’s likely that engaged employees are more likely busy “being engaged” in their job functions when a survey first becomes available. Since more of the early responses are from disengaged employees, there have to be fewer highly-engaged employee responses in the early returns. As time goes on, highly-engaged employees find the time to respond. Thus, the percentage of highly-engaged employees rises over time.
Conclusions
Since the percentage of highly-engaged and disengaged employees changes during the time an employee survey is open, it is important to minimize the effect of this on the survey results. We also have a goal of minimizing non-response bias, which is accomplished by driving the response rate to the highest-possible level. Therefore, it becomes very important to push very hard in the first survey period to obtain a high response rate, and to attempt to maintain that response rate in future surveys. A low initial response rate may understate the level of actual employee engagement. Obtaining a very high initial response rate* and letting it drop in subsequent years could result in a decline in reported engagement, even though a decline has not taken place.
This is not merely an academic exercise. One of the goals of employee engagement surveys is to detect changes in engagement over time. Not maximizing the response rate in each time period can cause engagement to appear to be rising or falling, even though it actually has remained stable. This can make it difficult to see the relationship between policy changes and employee engagement.
* Our average response rate is 80%. We consider anything less than 70% to be low and anything above 85% to be very high.