Join the Q: Chasing journal indicators of academic performance

Universities are predisposed to rank each other (and be ranked) by performance, including research performance. Rankings are not merely about quality, they are about perception. And perception translates nominal prestige into cash through student fees, block grants from government, as well as research income. As a consequence, there is a danger that universities may chase indicators of prestige rather than thinking about the underlying data that informs the indicator, and what the underlying data might mean for understanding and improving performance.

This image comes from an article published in The Conversation under a creative commons license. see https://bit.ly/2ExLDNB

Ranking has become so crucial in the life of universities that it infuses the brickwork and is absorbed by us each time we brush along the walls. At one time, when evaluating research performance, an essential metric was the number of publications. That calculus has shifted and it is no longer enough to publish. Now we have to publish in Q1 journals; i.e., journals ranked by impact factor in the top 25%. Those ranked in the next 26th to 50th percentile are Q2, and so forth. Unfortunately, Q-ranking encourages indicator chasing. It has a level of arbitrariness that discourages thoughtful choices about where to publish, and leads to such unhelpful advice as, “publish in more Q1 journals”.

The Q-ranking game was brought home to me in a recent discussion among colleagues in medical education research about where they should publish. In this discussion BMC Medical Education was identified as a poorly ranked journal (Q3) that should not be considered.

I like and entirely approve of publishing research in the very best journals that one can, and encouraging staff to publish in high-quality journals is a good thing. “Best” and “high quality”, however, is not just about the impact factor and the Q-ranking of a journal. The best journal for an article is the journal that can create the greatest impact from the work, in the right area, be that in research, policy, or practice. A personally, highly cited article in a low impact factor journal may be better than a poorly cited paper in high impact factor journal.

Some years ago I was invited by a government research council to review the performance of a university’s Health Policy Unit. One of my fellow panel members was very focused on the poor ranking of most of the journals into which this unit was publishing. The director of the unit tried to defend the record. She argued that it was more important that the publications were policy-relevant than that they were published in a prestigious journal. The argument was cut down by the research council representative. From the representative’s point of view, the government had to allocate funds, and the journal ranking was an important mechanism for evaluating the return on investment.

I did a quick back of the envelope calculation. It was true, the unit had published in some pretty ordinary journals — not an article in The Lancet among them. However, if one treated the collection of papers published by the unit as if the unit was a stand-alone journal, the impact factor exceeded PLoS Medicine, a highly regarded Q1 journal. My argument softened the opposition to refunding the unit, but it did not completely deal with it because the research council didn’t care about the individual papers. They wanted prestige, and Q-ranking marked prestige.

So, which medical education journal should you publish in? The advice was blunt. The university uses a Thomson Reuters product, Journal Citation Reports (JCR) to determine the Q-ranking of journals. BMC Medical Education ranks quite poorly — Q3 — so don’t publish there. The ranking in this case, however, was based on journals bundled into a comparison pool that JCR calls “Education, Scientific Disciplines”. This comparison pool includes such probably excellent (and completely irrelevant) journals as Physical Review Special Topics-Physics Education Research and Studies in Science Education. However, if one adopts the “Social Sciences, General” pool of comparison journals, which JCR also reported, BMC Medical Education jumps from a Q3 to a Q1 journal. And this raises the obvious question, what is the true ranking of BMC Medical Education?

The advice about where to publish explicitly dismissed an alternative source for the Q-ranking of journals, Scimago Journal Ranking (SJR), because it was too generous — with the implication that “generous” meant “not as rigorous”. In fact, it appears that the difference between SJR and JCR is about the pool of journals used for the comparison. Both SJR and JCR treat the pool of journals against which the chosen journal should be compared as a relatively static one. But it is not. The pool against which the journal should be compared (assuming one should do this at all) is dependent on the kind of research being reported and the audience. Consider potential journals for publishing a biomedical imaging paper. The Q-ranking pool could be (1) general medical journals, (2) journals dealing with medical imaging, (3) radiology journals, or (4) radiography journals, (5) some more refined subset of journals. As with BMC Medical Education, the Q-rank of prospective journals could be quite different in each pool.

One might reasonably, and rhetorically ask, did the value of the science or the quality of the work change because the comparison pool changed? This leads to a small thought experiment. Imagine a world in which every journal below Q1 suddenly disappeared. The quality of the remaining journals has not changed, but three-quarters of them are suddenly Q2 and below. (As an aside, this is reminiscent of the observation that half of all doctors are below average).

If a researcher works in a single discipline, learning which are the preferred journals to publish in becomes second nature. If one work across disciplines, then the question is not as clear. The question is no longer, “which are the highest ranked journals?”, but “which are the highest ranked journals given this article, and these possible disciplinary choices?”. If the question is, “which journal will have the most significant impact of the type that I seek?”, then Q-ranking is only relevant if the outcome sought is to publish in a Q1 journal from a particular comparison pool. If one seeks some other kind of impact, like policy relevance or change in practice, then the Q-rank may be of no value.

Indicators of publishing quality should not drive strategy. Strategy should be inspired by a vision of excellence and institutional purpose. If you want an example of how chasing indicators can have a severe and negative impact, have a look at this (Q1!!!!) paper.

 

Donald Trump’s BMI: getting the measure of the man.

I find myself fascinated by a pointless lie because it is inescapably tragic. All it can do is diminish the person in the eyes of others. And this brings us to Donald Trump’s height. In January 2018, the Physician to the President, Ronny L. Jackson MD asserted that Donald Trump was 6’3″ tall (1.90m). This is so unlikely to be true, that it stretches credulity. There is no reason for Jackson to lie spontaneously about a patient’s height, and it seems probable that he was encouraged to add a few inches by the President himself.

When asked to self report height both men and women in the US tend to overstate it.  Burke and Carman have suggested that overstating height is motivated by social desirability — you can never be too tall. There is ample evidence of Donald Trump’s (misplaced) search for the socially desirable with respect to his hair, his tan, his ethnicity, his intelligence and now his height.

In 2018 we learnt that Donald Trump was officially not quite Obese (body mass index (BMI) <30), and in 2019 he had nudged over the line into the obese range (BMI ≥30). Overstating height creates a problem in the calculation of BMI — which is mass (in kilograms) divided by height (in meters squared). Given that Donald Trump is likely shorter than 1.9m (6’3″), and probably closer to 1.854m (6’1″) this will have implications for whether he was really obese in 2018 (not just overweight as stated by his Physician) and just how obese he probably is (Figure 1).

Figure 1: Donald Trump’s BMI in 2018 and 2019 given different assumptions about his height [R-code here].

In 2018 Donald trump was just below the obese category if and only if he was really 6’3″ (1.9m) tall.  At any height less than that he was obese in 2018 and he is obese today.  His most likely true height given comparisons with others (cf, Barack Obama) is 6’1″, and this puts him comfortably in the obese range.

Misrepresenting one’s height does not create a problem if the lie is reserved for others — except perhaps in a political sense. Problems arise if one deludes oneself. Telling others that you are taller and healthier than you really are is one thing; if you lie to yourself you cannot properly manage your health.

 

 

 

Play with Big Tobacco and you will be tarred

Philip Morris International (PMI), profits by selling the world’s leading cause of preventable death — tobacco. The Foundation for a Smoke-Free World (FSFW) recently handed PMI a public relations coup by accepting a $1 Billion donation. Who now could credibly work with FSFW?

PMI is the world’s largest, international tobacco company.  It is quite explicitly not interested in a tobacco-free world and it works hard and secretly to subvert tobacco control. Its raison d’être is the sale of tobacco products, and the “smoke-free world” cover provided by FSFW looks like a Big Tobacco tactic in a long line of them.

There is little doubt that FSFW as an organisation has placed itself in moral jeopardy by accepting PMI’s money: “Moral jeopardy occurs when a person or an organisation attempts to do good using resources from a source that involves harm.” And here is the rub.  One of FSFW’s stated goals is to support global research through the support of “Centers of Excellence”.  Any research group, however, that accepts FSFW money is exposing itself to moral jeopardy. And like other health and medical research outputs from conflicted industry sources, the results cannot be trusted — no matter how genuine the researchers are in their belief of independence.

PMI’s money laundering scheme for researchers may provide a scent of freshness, but the tobacco tar will stick.

Beware of Big Tobacco bearing gifts

Two days ago it was announced to much fanfare that the international tobacco giant, Philip Morris International (PMI) pledged $1bn over the next 12 years to the Foundation for a Smoke-Free World (FSFW) to fund scientific research designed to eliminate the use of smoked tobacco around the globe. The Lancet editor, Richard Horton tweeted a challenge to the health community: how should we respond? My tweeted response is shown below, but the challenge drew a range of (usually negative) responses, and the whole announcement is worthy of further unpacking and analysis.

First, it is worth pondering PMI’s motivation. Would turkeys vote for Christmas? Would tobacco companies vote for a world without tobacco?  And the answer is no, they wouldn’t.

The announcement of the funding was picked up by, among other media outlets, Bloomberg, the Guardian, the Financial Times, Fortune, and CNBC.  Most of the headlines start with two words, “Philip Morris”, which means the corporate social responsibility team at PMI can expect a big elephant stamp on their performance appraisals and an end-of-year bonus. This kind of positive publicity for a tobacco company is extraordinary and under other circumstances I would have said they couldn’t buy it, but apparently they can.

If you visit the PMI corporate web-site you will be presented with a glossy video of talking heads with overlaid text that PMI is “Designing a smoke-free future”. Some time in the future PMI will be out of (less reliant on) their cigarette business — not their tobacco business, but their cigarette business.

We’re dedicated to doing something very dramatic – replacing cigarettes with the smoke-free products that we’re developing and selling. That’s why we have a total of over 400 dedicated scientists, engineers, and technicians developing less harmful alternatives to cigarettes at our two Research & Development sites in Switzerland and Singapore. It’s the biggest shift in our history. And it’s the right one for our consumers, our company, our shareholders, and society.

That is clearer.  PMI is still deeply committed to the manufacture and sale of an addictive, harmful substance, but in the future they hope it will be less harmful.  The level of PMI’s commitment to the shift away from smoked tobacco can be gauged by the allocation of 400 of their 80,000 staff to the innovation of smoke-free products, or about 0.5% of their workforce.  It should also be noted that when countries have attempted to interfere with PMI’s capacity to trade in a lethal product, it has fought those measures vigorously.  This means that PMI will fight extremely hard (as is required by its fiduciary duty to investors) to maintain its profitability even at the expense of the life and health of its consumers.  If PMI can develop a range of successful proprietary tobaccos products/technology that are smoke free, it is entirely in PMI’s interest to throw its weight behind anti-smoking initiatives, because competitors will be forced to use PMI technology under license.

Having got a sense of PMI, I will shift focus, more directly to the funding for FSFW.  In the light of the PMI pro-tobacco, smoke free agenda, ponder the name of FSFW.  This is a foundation dedicated to a smoke-free, but not a tobacco-free world. This is consistent with the commercial goals of PMI. For some, the argument is one of harm reduction. If the world moves to smoke free products, harm will be reduced. The fact that PMI is funding it, is proof of the excellence of public-private-partnership models.

Would it reduce harms? Yes.  The harms, however, may not be reduced to the extent that might be hoped. There are some indications that the use of smoke-free products carry an increased risk of mortality over non-smoking, non-use of smoke-free products.  Nonetheless, for those who would have become smokers, or switch from smoke to smoke-free products, harm will be reduced over being a smoker.

Does FSFW have a conflict of interest? They say, no.

Importantly, as established in the Foundation’s bylaws, PMI and the tobacco industry are precluded from having any influence over how the Foundation spends its funds or focuses its activities. Independence and transparency are core principles of the Foundation and all activities will be conducted with full transparency, free of tobacco industry influence. The Foundation has, constituted in its bylaws, an independent research agenda, independent governance, ownership of its data, freedom to publish, and protection against conflict of interest. Furthermore, strict rules of engagement will be put into place to ensure any interactions with the tobacco industry are fully transparent and publicly reported.

This is naive.

PMI has pledged about $80 Million per year over 12 years.  this represents about 0.11% of of PMIs annual net revenue. The money is not paid up-front or held in an escrow-type account, and the continued payment will be decided up by PMI — presumably factoring in PMI’s satisfaction with FSFW’s activities. And $80 Million buys a lot of loyalty.

The loyalty that funding from corporate giants garners was well illustrated recently by the New America Foundation (NAF). “New America is a think tank and civic enterprise committed to renewing American politics, prosperity, and purpose in the Digital Age. We generate big ideas, bridge the gap between technology and policy, and curate broad public conversation.”  Like FSFW they also boldly proclaim their independence from funders, stating under their Gift Guidelines:

New America steadfastly adheres to its mission of developing independent, non-partisan analysis and recommendations reflective of rigorous scholarship and promoting those ideas through broad public discourse. New America maintains full authority regarding project agendas, events, budgets, editorial content, and personnel decisions.

NAF receives substantial funding from Google.  One of the NAF Fellows commented (as part of his work) positively about a decision by a European Union antitrust regulator that went against Google. The employee was subsequently fired by NAF, after Eric Schmidt the Executive Chairman of Alphabet, Google’s parent company, spoke with the head of NAF and made his displeasure known.  This is instructive for two reasons, first Google was famous for its internal corporate motto “Don’t be evil” and later in Alphabet, “Do the right thing”.  If a company that had “do the right thing” as an explicit part of its self image can’t leave independent foundations to be independent, what hope does a tobacco company have?  Second, companies react negatively to things that they perceive as a threat to profit, without regard to the legitimacy or moral rectitude of the threat.

One would have to be foolish or naive to believe that one could be independent of the hand that feeds them. Even if someone at FSFW felt truly unshackled and unconstrained, the invisible thread of funding would moderate the independence of judgment.

Mitch Zeller the director of the Center for Tobacco Products at the US Federal Drug Administration regards this approach to harm reduction as a positive step forward, because Public Health has been “stunningly unsuccessful“, at selling harm reduction thus far.  Unfortunately, government and industry are often tightly bound, ensuring that harm reduction does not work unless, as in this case, the industry can continue to make money from a (less) harmful addiction.

If government wants harm reduction to be successful, then government needs to get serious about tobacco control. Ban all forms of tobacco advertising. Ban the sale of cigarettes in locations frequented by children.  Enforce uniform, plain packaging. Ban smoking in public places. Tax the product heavily.  Place all taxation revenue from tobacco products into health promotion, quit campaigns, interventions, and tobacco related research.