Less of a burden, but less fairness and accuracy: Some thoughts on the new ARC Discovery Scheme two-step application process
The Australian Research Council’s new Expression of Interest (EOI) process for Discovery Project (DP) funding might pose less of an administrative burden on researchers and universities, but it fails to address another key issue with research funding in Australia; fairness. In fact, the new application process, a two-stage process with an initial EOI stage which doesn’t involve specialist peer reviewers, appears to be less fair and accurate than the process it will replace. A basic peer review process involving aggregative decision making methods at the EOI stage would address both the administrative burden of the single-stage grant process and enshrine a fair process.
In April of this year, the government released a review of the ARC, highlighting the challenges posed by the significant administrative burdens embedded in the grant application process. The existing procedure demanded considerable time and institutional resources from researchers, resulting in extensive applications for a grant program with a success rate of only 16.4% of the 2592 submissions in the latest round. In response, the ARC has announced the implementation of a two-stage process for its 2025 DP Scheme, with applications closing in February 2024. Under this new process, applicants will submit a concise Expression of Interest (EOI) for an expedited initial review before potentially submitting a full application. Notably, in contrast to the previous scheme, where all applications underwent peer review before assessment by the College of Experts, the EOI stage applications will be exclusively assessed by members of the College of Experts.
While the two-stage process addresses the challenge posed by the administrative burden of a single-stage process, the absence of peer review in the EOI stage presents a notable challenge to the fairness of the grant assessment process.
In framing the ARC review, the authors emphasized the necessity of reforms to "enshrine the importance of academic expertise, peer review, and transparent, fair competitive assessments." The new DP assessment approach jeopardizes these principles, however, as the (albeit expanded) College of Experts comprises just 269 senior researchers and the vast majority of applications will be rejected (the ARC has not said how many, but the ARC review suggested an EOI process with just 10% success rate). They cannot possess the required expertise to assess all applications on equal and fair grounds, there being a reasonable expectation that they will be unable to adequately assess applications beyond their area of expertise on their merits. To illustrate, my own discipline of philosophy, the current Field Of Research (FoR) code does not encompass the breadth of work conducted in philosophy departments, including applied ethics, philosophy of science, political philosophy, and mathematical logic. Even within the philosophy FoR code, the discipline is highly diverse, covering topics ranging from the nature of consciousness to the truth of moral theories. There are currently two philosophers in the College of Experts. They are highly regarded in their specific sub-fields, of course, but they would not claim expertise in all areas of philosophy. While all EOIs will be assessed by three members of the College, there being just two philosophers on the College poses a challenge to fair assessments of work across the breadth of the discipline. Philosophy is not an outlier. All disciplines are heterogeneous at the six-digit FoR Code level and have just one or two representatives on the College. This problem arguably is only made worse for interdisciplinary projects which require expertise in multiple fields.
In response to these concerns the ARC could further expand their College of Experts, but this approach still seems a blunt means to ensure a fair assessment of all applications. Rather, I argue, the ARC should draw on research in aggregative decision making showing that in situations with poor or limited evidence and high judgment variability (as would reasonably expected from non-specialist assessors), aggregative judgment processes tend to yield more consistent assessments. Requesting simple scores of feasibility and track record from multiple reviewers for each EOI in their broad area of research alongside a self-assessment of the reviewer’s confidence in the score, and then aggregating them with a weighted metric, would provide a more equitable and accurate basis for deciding which applications proceed to the next stage for full peer review.
(Thank you to Brian Hedden, Daniel Stoljar and Colin Klein for their discussions on this)
I've been toying with doing a podcast on science and philosophy for a while and the opportunity came up to do it for a course I teach, PHIL2126: Science in Society: Ethics, Public Policy & Scientific Practice. I'm recording short (10-20min) accessible intros to various topics for the unit and have made them in podcast format for broader consumption. Subscribe with your favourite podcast app here or listen to the first episode on the ethics of animal experimentation below! Enjoy!
A version of this blog post was published at The Conversation on 29/12/20
“It ain’t what you don’t know that gets you in trouble. It’s what you know for sure that just ain’t so.”
Over the course of the COVID-19 pandemic we have seen a deluge of outright lies, conspiracy theoriesand pseudo-sciencefrom various peddlers of self-interest. As a philosopher, more vexing than these calculated examples of misinformation has been the sloppy reasoning evident in public discourse on the international crisis. Every day, the basic failures in critical thinking that I teach first year philosophy students to avoid are being made by politicians, government officials, commentators and the general public. Although these simple errors in reasoning can be employed to deliberately mislead, it is more frequently the well-intentioned that fall victim to their appeal. The only antidote is a large dose of skepticism and some understanding of where our reasoning frequently goes wrong. Here are just three of the common errors to look out for:
1. We are not in Kansas anymore Toto
In arguing against various public health measures, particularly “lockdowns”, it is not uncommon to hear the drops in cancer diagnosesor the negative impacts of school closures (particularly onlow SES students) cited as the “hidden cost” of COVID. Whilst it is reasonable to ask whether the costs of lockdown outweigh the benefits, the costs that failing to impose a lockdown would impose rarely feature in this calculus. Rather, some kind of “pre-COVID normal” is the assumed baseline of comparison. This is a mistake. The rates of cancer diagnosis or school outcomes in pre-COVID times are irrelevant when thinking about the impact of public health measures in our current circumstance. What is relevant is the expected rates and outcomes given the impact of the COVID-19 infections that would occur without the public health measure in place. In the case of cancer rates, for example, we should anticipate rates of diagnoses to be lower than pre-COVID normal both with, and without, lockdowns in place. People are going to avoid going to the doctor in times of a pandemic even if it is easy to do so to reduce their risk of infection. Similarly, when looking at the negative impact of school closures on low SES students, the appropriate comparison must be with the negative impacts that more widespread COVID-19 infections are likely to have on those students. As has been shown both at home and abroad, the impacts of COVID-19 outbreaks are disproportionately felt by lower SES communities and it is reasonable to expect that this will disrupt education in these cohorts too. It is important to look at the costs and benefits of public health interventions but it is important to not to forget that without them, we are looking at the costs of more widespread infection, not “pre-COVID normal”.
2. The Devil is in the detail
In Australia, the overall survival rate for breast cancer in females is 91%. Upon hearing this, it is natural for those newly diagnosed with breast cancer to infer that this means that they have a 91% chance of still being around in 2025. Unfortunately, this is not really the case. For any given individual the details really matter. If the cancer has spread beyond the breast, for example, the survival rate drops to 80%. Factors like age, type of breast cancer and other factors also can make a marked difference. For any individual cancer sufferer the 91% figure is not really all that informative at all. The details here really matter. The same thing is true for the data surrounding COVID-19. Bald numbers hide significant details. NSW and Victoria are illustrative here. In comparing the NSW and Victorian public health responses to the “second wave” the conclusion that the more promising numbers in Sydney as compared to Melbourne must be due solely to differences in contact tracing approaches has been drawn by many, including the PM. This claim relies, however, on an overly simplistic view of the data, ignoring other important differences between the Victorian and NSW situation. Even if we assume equal contact tracing capacity between the two states, differences between the two states offer some reason for one to have had more success in controlling numbers than the other. For example, despite similar absolute case numbers over the ten days to October 14th, approximately 60% of the cases in NSW were due returned international travelers and it accounted for none of the cases in Victoria.Given that a positive case in a traveler in hotel quarantine is easier to contain than one in the general populace, it is natural the situation in Victoria to be more challenging than that in NSW. Similarly there are other features of the demographics of the Victorian outbreak that also set it apart from NSWsuch as the average size of the households in which infected individuals live and the source of their infections. Again, the bald numbers can mislead, the devil is in the detail.
3. Bad luck and chance
The Ancient Greeks blamed unexpected bad outcomes in their lives on Tykhe, the Goddess of chance. The Romans would similarly blame Fortuna. In our controlled modern world, however, we typically assume a bad outcome to be the sign of failure rather than bad luck or chance. In a pandemic such as this one, however, not only can relatively small differences between situations lead to large differences in outcomes, but these small differences often come down to luck. This is especially true when talking about very small numbers of cases such as we have in Australia right now. At such low numbers, bad luck and chance are going play a big role in our fortunesand it helps to keep this in mind. It is easy, for example, to think that any jump in case numbers is indicative of a failure of the public health measures in place. This is a mistake. Whether an COVID-19 positive individual lives with one other person or six, or whether they work in aged care, or from home, can make a significant difference to the potential number of others that they infect with the virus and the potential outcomes of those infections. It is much harder, for example, to contact trace all the people someone working outside of the home has come into contact with compared to someone working from home and only leaving to go to the shops once a week. Thus, no two infections are really equal. This doesn’t mean that jumps in numbers should not be concerning but it also doesn’t mean that it warrants any shift from our current public health measures. Fluctuations are to be expected due to the role that chance and luck unavoidably play in a pandemic situation.
Better safe than sorry? How much caution is too much in thinking about animal sentience and animal welfare?
I had a really enjoyable discussion with Simon Lauder about this topic on ABC South East NSW earlier this week (Wednesday 19th June 2020). The audio is available here.
The question of animal sentience (i.e. the capacity to experience pain) is fundamental to discussions of animal welfare—undue animal suffering is immoral and must be avoided and if animals. Whilst this much is agreed upon, applying this principle requires an understanding of animal sentience Which animals are sentient? Which are not? When is a given animal in pain or suffering? When is it not?
Answering these questions is often less than straightforward. Behaviours we associate with pain in ourselves, not always being present in animals in similar situations (e.g. an animal may conceal pain rather than yelp or cry out). In the face of this uncertain evidence, policy makers typically adopt a policy of “erring on the side of caution” or “giving the animal the benefit of the doubt”. This results in animal welfare policies which treat an animal as though it is capable of experiencing pain despite uncertain evidence (see discussion by Jonathan Birch here).
Whilst this approach has the great benefit of minimising unnecessary suffering in the world, recent of pain perception in bees and fish challenges the practicality of the principle. For example, worldwide some 970 to 2700 billion fish are wild caught annually. If fish are sentient, then the number of sentient beings in the form of fish that are slaughtered for food annually equals at least twelve times that of the current human population (see Bob Jones on this here). . This offers a great case for ceasing animal fishing on the grounds of being better safe than sorry morally. Such a policy would, however, come at huge human cost. 3.2 billion of the world’s population rely on fish for a significant proportion of their daily food intake, not to mention the economic reliance on fisheries of people This creates a dilemma for the welfare policy maker. Perhaps even more stark is the challenge with invertebrates like bees and flies (see discussion of bee sentience by Colin Klein and Andy Baron here). If the evidence of their sentience is correct, then there are serious moral implications of pesticides. Again, however, “erring on the side of caution” and avoiding invertebrate deaths and suffering would have massive human costs. How should we weigh up the evidence in such a situation? Should we abandon the principle of erring on the side of caution? Or should we bite the bullet and treat even the tiniest fly as sentient? Is there a middle ground?