How level was my playing field?I don’t necessarily expect everyone to be excited at the news that the 2013 Application Performance Monitoring Magic Quadrant assessment is under way. It’s a specialist field, and only a few dozen companies will be directly involved.

But this year, this is one MQ that looks like raising vital issues for every IT vendor that wants to be featured in the relevant Magic Quadrant – or that has ever worried about how even-handedly Gartner treats clients and non-clients.

In the 2012 APM Magic Quadrant, 14 companies were rated, two of them for the first time. Some 17 other vendors had been dropped since the 2011 assessment.

This was a massive shift in focus from year to year, with the inclusion criteria changing quite abruptly, apparently because of the arrival of a new analyst on the Gartner team, Jonah Kowall.

Suddenly, it seemed, more than half the previous year’s MQ participants were no longer relevant for APM buyers. We spoke to several of these companies and there was a lot of dissatisfaction about the sudden changes and the impact they might have on each vendor’s sales pipeline and 2012 business performance.

Criteria for customers only

Yesterday (February 28th), Jonah Kowall tweeted about the impending 2013 assessment process.

No doubt there are at least 31 vendors keen to review the survey for this year’s assessment and see whether they have a chance to participate and shine. But how level is the playing field for these players – and for all those others that believe they should also be part of the assessment?

Well, part of the answer lies here, in Jonah’s blog entry from 22 February.

What this blog discusses are the details of the assessment criteria for 2013. In fact, Gartner has published detailed evaluation criteria, with a useful explanation of how they have changed and how greater transparency has been created in the process for grading products.

That would be excellent news – except that all this information has been published as a research note and is available only to Gartner customers! And, as if to rub it in, Jonah goes further and writes “Sorry, clients only. Contact me via Twitter @jkowall if you need something.”

As a former Gartner analyst and executive for many years, I have always been among the first to defend the Gartner stand on impartiality when it comes to vendor assessments.

But this flies in the face of that principle. This is not impartial treatment of all relevant vendors.

By revealing the criteria to clients only, Gartner has stuck a flag in the ground that clearly waves the message “Yes, we are biased toward our own clients.”

While the blog entry provides a brief summary of the note’s contents, the fact is that by making such a huge and direct distinction between clients and non-clients when it comes to preparing for assessment, Gartner is blowing a gaping hole in its own claims to impartiality.

The big question

So here is the big question.

What is the Gartner official position on this approach?

Will Gartner release the criteria to every interested vendor, immediately, by providing a link to the full document on the blog?

Or is this the beginning of an official recognition that if you want to be included, and ensure that you can focus on providing the right information to be assessed equally, you must pay to be a Gartner client?

Gartner, it’s over to you.

Update 3-3-13

Following the posting of this blog on Friday 1st March Gartner were quick to respond. Three hours from our blog posting, Jonah’s blog was modified to explicitly state that all companies, not just Gartner clients, are free to participate in any Magic Quadrant.

However as this statement did not in any way address the inequality in information to participants we posted a further response directly on Jonah’s Gartner blog clarifying our view that the playing field must be levelled if Gartner are to continue to reasonably claim impartiality.

Two hours later Jonah’s blog was amended again to specifically state that, whilst the online report remains client access only, non-clients should contact Jonah directly to receive the criteria.

This statement does still fall short in terms of absolute clarity for non-clients but we applaud the Gartner rapid response and efforts to maintain their position of impartiality when it comes to the ability for all vendors to participate in Magic Quadrant assessments, irrespective of client status. We would like to have seen Jonah explicitly state that he will send the criteria research report to any firms interested in participating in this assessment – but we trust that this is what is implied.

If you are an APM vendor who is not a Gartner client – we recommend you contact Jonah as suggested (FYI, as Jonah invites email without sharing his contact details, the naming convention for Gartner email addresses is lastname.firstname) and request the report. Please do let us know if you are not provided with this information or would like our assistance.

For other non-Gartner clients we recommend keeping a very careful eye out for errors like this in the future. We believe that this was a genuine mistake versus a deliberate intent to bias the assessment process – but if the error can occur once, it can occur twice. If you see a similar situation we recommend immediately alerting the Ombudsman at Gartner to ensure that your firm has as much information to help prepare for an assessment as any other.