Money in envelopeWhen considering whether to sign a contract with Gartner, most companies are perfectly well aware of the two hats that Gartner wears when engaging with them – and of the potential conflicts of interest this could create.

In one of these roles, Gartner helps its clients as their “strategic adviser”. But when it is wearing its other hat, it acts as the “formal assessor”.

In an ideal world, decisions about whether to work with Gartner should always be based on the former. In reality, though, most are based on the latter. Many companies choose to sign up because they believe that handing over their money and becoming Gartner customers means they will positively impact the analysts’ view of them in the formal assessor role.

Mythbusted

We’ve heard it all before – and so, probably, have you. In its recent filing suing Gartner over what it saw as an unfair Magic Quadrant assessment, NetScout spelled out the allegation that Gartner profited by this confusion of roles.

“Gartner has a ‘pay-to-play’ business model that, by its design, rewards Gartner clients that spend substantial sums on its various services by ranking them favorably in its influential Magic Quadrant research reports and punishes technology companies that choose not to spend substantial sums on Gartner services” said NetScout’s legal submission.

Or, to put it more bluntly, in the words of the CEO of one of our clients, “The Magic Quadrant process is either black magic or blackmail.”

This is a key issue for our industry, so we decided to see what we could discover from a detailed analysis of our own clients’ results over the last two years.

During this period, The Skills Connection has worked with some 60 different clients on Magic Quadrant assessments. Most of these companies, but by no means all, are also Gartner clients.

The upshot will surprise the cynics. Within our, admittedly small, sample, what our research showed is that there appears no direct correlation between simply having a financial relationship with Gartner and achieving a positive position in a Magic Quadrant.

Examples from the front line

Let me share two examples to illustrate this, based on two recent MQ assessments for Skills Connection clients that were not Gartner customers.

In the first MQ, there were 18 vendors featured in the assessment; in the second, there were 17. The illustration highlights the most significant movements from year to year, from 2013 to 2014.

First MQ assessment example

do you have to be a client image 1 25-11-14

Second MQ assessment example

do you have to be a client image 2 25-11-14

Clearly, our clients did well in the assessments. Both of them improved their positions considerably, despite their lack of any formal relationship with Gartner. In the first MQ, four other vendors also dropped significantly. Based on who they are, we believe that at least three of these four were likely to be Gartner clients. In the second MQ, we believe the two vendors whose positions deteriorated were probably Gartner clients as well.

I wouldn’t claim this was an exhaustive study, but it is a valid contribution to the argument. On the basis of our sample of 60 MQ contenders, we believe there is evidence that a financial relationship is not vital to achieving MQ success. Nor, for that matter, is it any kind of guarantee.

But then again, it’s not exactly a level playing field

That is not to say, though, that being a Gartner client cannot tilt the playing field to a customer’s advantage. In our earlier blog post “Can you spend your way to the top right corner”, we discussed how client privilege enables customers to get themselves more analyst engagement time and make the engagement a two-way conversation, in order to build a mutual understanding. For companies that use their Gartner membership judiciously, there is a real opportunity to tilt the playing field. But we see this as being related to activity, rather than the cash payments involved.

Engage, sulk or sue?

Netscout is not the first firm to sue Gartner, nor will it be the last. But counting on the legal route as a strategy for achieving a good assessment is extremely expensive and highly risky.

It’s your call, of course. But our best advice would be do what our two example clients did above. And by that I don’t just mean “Hire The Skills Connection to help”, though I would certainly welcome that, too.

What I do mean is that you should focus on engaging as effectively as you can, irrespective of your formal relationship with Gartner. Plan ahead, put in the effort and make sure your story is clear. Concentrate on what the analyst needs to know and see that your case is put across in a compelling and consistent way over a period of time, backed up by factual evidence at every possible point.

At the end of the day, being a Gartner client is no guarantee of a good assessment result. But, equally, not being a client is no guarantee of a poor one. Whichever route you choose to take, the key is simply to spend your money and your time wisely.