Answers From Jupiter Research (JupiterKagan)

07/11/2006

Back-story
A few weeks ago JupiterKagan/Jupiter Research issued a press release that provided results from a study about corporate blogs - Jupiter Finds That Deployment of Corporate Weblogs Will Double in 2006. I was puzzled by one particular finding -  "... nearly 70 percent of all site operators will have implemented corporate blogs by the end of 2006." The stat seemed high to me.

I contacted JupiterKagan/Jupiter Research, via their agency Peter Arnold Associates, to lean something about the methodology (not included in the release). JupiterKagan/Jupiter Research refused to provide additional information (which in all fairness was their right).  However, concerned with the study's conclusions, I voiced my questions - which I felt was my obligation to the readers of Diva Marketing. Here's the link to the Diva Marketing posts one and two and three.

The posts caught the attention of many in the biz blogosphere including Fard Johnmar who bought the research and came to his own conclusions; and Ian Betteridge who commented on NevilleHobson.com that I should have contacted Jupiter directly. Ian I took your advise.

JupiterKagan's/Jupiter Research's Response
I left a voice message for David Schatsky, President, JupiterKagan, Inc. I offered Mr. Schatsky an opportunity to tell his side of the story. The following is the unedited eMail exchanges that Mr. Schatsky kindly gave me permission to post on Diva Marketing.

David Schatsky's eMail response  #1

Hi Toby.

Thanks for your voice mail. I appreciate your interest in our research.

You must be aware that it is disingenuous in the extreme to publicly voice "concerns about JupiterResearch's lack of methodology information to support the findings of a study" when in fact our methodology is transparent to all of our clients.

We do not tend to devote time to providing information of this type for free to anyone, as it's generally beyond the scope of the interest of publications who follow the sectors we cover and does not benefit Jupiter or Jupiter's clients--who pay for the privilege of exclusive access to this information. You owe your readers a correction of this misrepresentation.

I'll tell you that some of the data cited in the report you are discussing and mentioned in our press release is from a survey of 251 executives from a variety of industries who make decisions about their company's Web site spending and who work at companies with $50 million or more in revenue.

It's worth noting that some of the other bloggers you engaged in dialog thus misrepresented the focus of our study, suggesting it focused on Fortune 500 companies. That's not what we said in our report or in the release. For the sake of your own credibility, it's worth correcting that misrepresentation as well.

Thanks and best regards,

David Schatsky
President
JupiterKagan, Inc.

Toby's eMail response:

Mr. Schatsky -

Appreciate your email response to my message. 

I want to make it very clear that my intention was never to bash JupiterKagan; however, when an organization provides research findings to the general public, for example, as stated in a media release, without the benefit of background details don't you agree it may cause people to question how those findings were determined? Although JupiterKagan's clients may be familiar with its methodologies,  I'm sure JupiterKagan doesn't expect the marketing community at-large to accept results from (any) research at face value.

In my message to you I offered the opportunity to present your side of the story. Would you like me to post your response to Diva Marketing? I will  be happy to publish your comments unedited. If you prefer that this remains a private conversation between us I will  certainly honor your request.

Looking forward to your response.

Very best.
Toby

David Schatsky's eMail response #2

I agree that our news releases may cause people to question how our findings were determined.  That's a good thing from our perspective. An aim of those releases is to motivate people that have a significant stake in understanding and applying out findings to inquire with us about becoming clients. 

JupiterResearch clients have unfettered access to our analysts. Our analysts can not only explain the finer points of methodology, but also provide market intelligence, advice on best practices, insight into trends, and so on.

Feel free to post my response on your site.

Regards,
David Schatsky

Mr. Schatsky seems to think that I owe you an apology. I apologize to all readers who feel I misled them. I stand firm in my belief that any marketing research firm that offers results to the public should include information about the methodology. If not, don't be surprised if marketers, bloggers, the media or prospective clients question your findings and draw conclusions based on the information provided including questioning the credibility.

Sidebar: The debate for the marketing research community - Is it a business decision not to include methodology with publicly published findings or is it an ethical issue? Thanks to Bill Neal, SDR, Inc. for the link to the AAPOR Code of Ethics - reference - III. Minimal Disclosure of Research Findings (American Association for Public Opinion Research).

Technorati Tags: , , , , , , ,

Trackbacks

Trackback url:
http://www.typepad.com/services/trackback/6a00d83451b4b169e200d8345f878669e2

» The twisted Jupiter tale from NevilleHobson.com
Id intended to title this post more twists in the Jupiter tale but I think the actual title is far more appropriate. Quick recap: Last month, Jupiter Research published a study about corporate blogging that led to Toby Bloomberg a... [Read More]

Tracked on Jul 12, 2006 9:54:49 AM

» The JupiterKagan Corporate Blogging Survey: Four Lessons I Learned From This Incident from HealthCareVox
The HistoryOn June 29 Toby Bloomberg, author of the Diva Marketing Blog, posted an article questioning the results of a JupiterResearch (now JupiterKagan) study. A press release distributed by the firm seemed to indicate that a significant number... [Read More]

Tracked on Jul 13, 2006 12:36:18 PM

» Power to the Bloggers from Results Revolution - Getting Real Marketing Results for Small Business
Last week, the hot topic among the business blogging community was a JupiterResearch study report entitled: Corporate Weblogs: Deployment, Promotion, and Measurement. Read the original story thread at MarketingDiva's blog. The bottom line was the Jupit... [Read More]

Tracked on Jul 17, 2006 10:23:31 PM

» JupiterKagan Hits A Snag Over A Name from HealthCareVox
A few weeks ago, JupiterKagan ignited quite a bit of a blog storm about a study it conducted focusing on the use of blogging technology by corporations. Now the firm has raised the ire of a different group of bloggers... [Read More]

Tracked on Aug 31, 2006 1:37:56 PM

Comments

Toby,
I think that if the purpose of the press release was lead generation...to encourage people to become Jupiter Clients and was not meant to be considered research results, thus not bound by the AAPOR code of ethics; then the press release should have stopped at saying that the report on corporate blogging was available. Since the press release provided research results without sufficient methodology, your inquiries and the time that you have put into this were not only
justified, but valuable to everyone who might wish to use the data.

Furthermore, since Fard Johnmar bought the report that still didn't provide methodologies, it seems that Jupiter should re-evaluate their position on the release of this information as well as future releases.

Thanks for all the time that you put into this.

Posted by: Marianne Richmond on Jul 11, 2006 3:31:40 PM

Toby, contrary to what he says, I don't think you were the one being disingenuous! His first response basically says only paying clients get to know where the numbers come from. So this is a sales pitch, not news. Fine, then don't issue a press release -- just share the results with your clients! I believe to put out a release for the media touting results with no explanation of methodology -- and then to refuse to give it when asked -- is what is disingenous. It's SOP to explain HOW you came to the conclusions you are publicizing. As a former journalist, I know this to be the case. Stick to your guns, Toby, you have been completely responsible!

Posted by: Katya Andresen on Jul 11, 2006 3:53:19 PM

That's also what concerns me about Mr. Schatsky's response... Fard Johnmar bought the report, and didn't get the methodology included in it. While this in and of itself may break with the AAPOR code of ethics, his conclusions, including information he gathered from conversations with JupiterResearch and their represenatives, left him feeling the methodology and the information he gathered about it are not only incomplete, but wholly inadequate.

This is still an issue for me, and all David Schatsky did was gloss over it.

Posted by: Eric Marden on Jul 11, 2006 5:23:40 PM

I can understand Jupiter not wanting to share detailed results of the survey -- that is their proprietary information. But if they are going to release pieces of it to the public, they need to be prepared to back up the credibility of the data by answering basic questions about their research design. The fact that they will not share information about the validity of the data throws up red flags to me. For a research company not to be willing to provide a short summary of their methodology (beyond the vague description provided by Mr. Schatsky) makes it seem that they are not confident enough in their methods to expose them to public perusal. If I were a potential client, this evasiveness would turn me off rather than make me want to sign up to find out more.

Posted by: Nedra Weinreich on Jul 11, 2006 6:14:12 PM

Toby my friend, you are on solid ground IMO. The email from Jupiter makes it even more clear that they are merely using "research" as a tease to get clients. You don't owe anyone an apology.

Posted by: Susan Getgood on Jul 11, 2006 7:02:47 PM

Toby, I believe you were filled full of *pure* genious when you pushed for more information on this research. Mr. Schatsky's response left a bad taste in mouth, like that of a company serving only its high paying corporate clientele. To release information to the public, then be unwilling to back it up, whether it is their right or not, not only puts a big question mark on the research, but patronizes those who are only looking for simply answers. How dare anyone question the research? Afterall, we are expected to smile and accept. Not question. Good for you.

Posted by: Chloe DiVita on Jul 11, 2006 7:06:15 PM

Dear Toby,
As the author of the recent study, Behind the Scenes in the Blogosphere:
Advice from established bloggers(www.umassd.edu/cmr), I too was very interested in the methodology used by Jupiter. Thank you for pursuing the question. While I respect Jupiter's right to generate leads, and choose what to disclose, I think the press release was the wrong way to go. One of the many things I've learned in the course of my research is that transparency is highly respected in the blogosphere. I think Jupiter would have gotten more respect and more clients by openly sharing their information including their methodology.

Posted by: Nora Barnes on Jul 11, 2006 7:07:50 PM

As I see it, Toby, you have donre a fine job in revealing another naked Emeperor. While I advocate transparency, this guy makes it so that you can see right through him. In short he put out a bunch of titlating data, not to educate the public, not to add to an ongoing conversation but to tease potential customers into paying to learn more. That's really lame.

Posted by: shel israel on Jul 11, 2006 10:51:05 PM

Is it any surprise that I agree with you and your actions? I have to agree with the others here who point to the fact that the Press Release created a belief that information was being SHARED as opposed to SOLD. It wasn't a press release, but rather an advertisement. As Shel said; "lame".

Good for you- you done good.

Posted by: Tim Jackson on Jul 12, 2006 2:10:33 AM

Toby..
"Mr. Jupiter" is on Mars. As a journalist for 32 years I can tell you that no credible organization publicizing its research via news release would omit or refuse to divulge its methodology. Even a rookie reporter knows their story has no chance of being published without including information on how a study was carried out. A company that refuses to include that information will find their news releases are quickly used to line waste baskets. Come back to Earth, Mr. Jupiter. The fresh air might clear your thinking.

Posted by: Ed Garsten on Jul 12, 2006 9:06:07 AM

This is an interesting wrinkle - here what Seth Godin had to say about another Jupiter study:
http://sethgodin.typepad.com/seths_blog/2005/03/file_under_stat.html

Posted by: Katya Andresen on Jul 12, 2006 11:50:29 AM

Hi Toby,

I don't think you owe anyone an apology, questioning the research methodology in a space where you interest and expertise seems perfectly legitimate to me. This is an excellent post, good research.

John

Posted by: John Cass on Jul 12, 2006 11:35:25 PM

Thanks all for your support. I don't believe I did anything wrong either. My hopes are that the next time marketers see research in a partial form e.g., a media release or in a presentation we will not blindly accept the findings without an understanding of how those numbers were derived.

In looking around at a few other firms it appears that JupiterResearch is not alone in not providing information for us to make intelligent decisions. Perhaps if we continue to demand that AAPOR standards are met, not post half-baked findings and call out firms that do not meet the Code of Ethics we can positively impact our profession.

Posted by: Toby on Jul 13, 2006 12:23:09 AM

Post a comment