Author Archives: Dr. Nicole Barbaro

About Dr. Nicole Barbaro

Nicole holds a Ph.D. in psychology and is currently a Research Scientist at WGU Labs

An Intimate Look at What’s Not Working in American Higher Education

I’ve spent my entire adult life thus far at university. I began my undergraduate degree at 17 years old and just graduated with my doctorate at 28. And, I now work at a non-profit education innovation hub affiliated with the largest university in the United States. I have devoted so much time to my education personally, and now professionally, because I love learning and, quite honestly, I am not sure what else I could do or would want to do.

Maturing personally and professionally within the American higher education system has left me with seemingly conflicting views of our institutions that I’m sure many other higher education professionals and professors share. On the one hand I view education as one of the most, if not the most, important thing in one’s life; education and learning enable everything else within my life. On the other hand, there are very real problems with our higher education institutions today, problems that are leaving students behind and underserving too many students once they get to campus.

The Years That Matter Most: How College Makes or Breaks Us by Paul Tough is the most compelling account of the myriad problems plaguing higher education that I have read. Aside from what I think is an utterly uncompelling title (one that really undersells the importance of the book in my opinion), the data and stories contained within its pages give broad, yet intimate insights into the lives of the students that our education system is failing in mass numbers.

What makes this book special and enormously moving are that the problems with our higher education system Tough details are not only backed by data, but demonstrated by real stories with real college students that Tough interviewed and followed over many years. Connecting abstract institutional problems with the lives of real students makes the problems in higher education tangible and salient.

Tough organizes his reporting somewhat chronologically, starting with the decision to go to college and how it differs across income level. For students in higher income levels, the general rule and decision-making process of which college to attend is to go to the most selective college that will accept you. For lower income students, many of whom have parents without college degrees, figuring out which college to attend typically isn’t based on selectivity, but rather location or unambitious recommendations from high school guidance counselors. In what Tough calls “income-typical behavior” high-achieving students from low income neighborhoods miss out on the advantages that going to a selective university can offer for their economic mobility.

Tough then dives into the big business of test prep, and the importance of the SAT/ACT scores in higher education. The decision to heavily weigh SAT/ACT test scores, or use them at all, in the college admissions process is a hotly debated topic. On the one hand, the test scores are the most predictive piece of information admissions offices have for whether a student will be successful (i.e., GPA and graduation likelihood). On the other hand, the use of test scores appears to be leaving a non-trivial minority of students behind. And much of the time, these students are from below-average high schools that tend to be in low-income and/or minority neighborhoods.

College admissions and its focus on test scores works against many high performing students. There is a group of students, which Tough refers to as “deflated-SAT” students that have test scores much lower than their high school GPA would predict. These students account for about 1/6th of those that take the SAT/ACT. For the remaining students that take the SAT/ACT, 2/3rd of them score consistent with their GPA and the other 1/6th score higher than what their GPA would predict, who Tough calls “inflated-SAT” students. The students in the deflated-SAT group compared to the inflated-SAT group, however, are three times more likely to be low-income, three times more likely to be first generation college students, twice as likely to be female, and at least twice as likely to be black or Hispanic. With the exception of women, then, test scores work against already underserved populations in our institutions.

Tough describes research by DePaul university, which is a test-optional institution meaning that students with deflated SAT scores are not required to submit them if they choose not to, shows that students who do not submit their test scores do, in fact, have ACT scores about five points lower, on average, than students that do choose submit them (the university requests that they submit their scores for research purposes). Importantly, however, these students with deflated scores are no different than the students who submit their scores: they have equivalent freshman GPAs, are just as likely to enroll in their sophomore year, and their six-year graduation rates are within two percentage points of the test-submitting students.

After getting in, students need to stay in. Tough’s chapter, Staying In, describes many big problems facing public universities today. How to best help an increasingly diverse student body, and how to get students to graduate. At most public state universities, only around two thirds of students actually graduate, with rates being far higher at more homogeneous Ivy league universities and other prestigious institutions, but much lower for non-selective institutions and community colleges. Thus, even at our best state institutions, too many students are unsuccessful. And, these problems disproportionately plague minorities (but not Asians), first generation students, and low income students.

Why this happens is due to many reasons. For example, primary and secondary educational institutions are not equal across neighborhoods, and the prior knowledge and education of incoming students varies dramatically. In many ways, problems in higher education are rooted in many of the problems plaguing our primary and secondary schools. Moreover, underserved students are often the minority on campus, and their sense of belonging and community can suffer. These big problems are difficult to address at scale, though Tough describes inspirational cases of programs, such as those within the University of Texas university system that are making strides to combat these problems.

The Years That Matter Most is truly eye opening to higher education’s problems; but isn’t ideologically driven. The stories he shares about real students facing real challenges at their university offer something for many to resonate with. And, Tough made me reflect on the ways in which these problems have personally impacted me during my decade-long educational journey. Although higher education does, indeed, work just fine for most students, our higher education institutions need to work for more. We need to find ways to build up underserved students – minorities, low income, first generation, and deflated-SAT – and we need to do so at scale.

Photo by Nathan Dumlao on Unsplash

It’s Not You, It’s the Market

Peak interview season has passed and you’re starting to see all the “I’m excited to share that I’ll be joining the Psychology Department at University X this Fall” tweets on your feed. Although exciting for those that beat the odds and landed a coveted tenure-track position, the reality is that the majority of psychology PhDs will not be tweeting out an incoming faculty announcement anytime soon. Why? Because, as most of us are aware, the market is, well . . . fucked. Seriously, it’s not you, it’s the market.

The intuition that landing a tenure track faculty job comes down to mostly luck and how many tickets (applications) you buy is beginning to be backed by data. As interest in meta-science research has taken off in recent years, so too, has research and analysis into the academic job market. What is the best predictor of landing a job? Submitting a lot of applications. Really, that’s it. The more darts you throw at the board, the better your chances of hitting the bullseye.

During my early years as a graduate student, I knew it was normal to submit upwards of 100 applications, and maybe get a handful of interviews. I knew, also, that it was normal to spend several years adjuncting and post-docing on insufficient pay waiting to land a faculty role.

When I started graduate school, I was determined to beat the odds — and I worked. In my five years I published more than 30 peer-reviewed research articles (but 42 scholarly articles total), I gave more than 25 conference presentations, got elected to the board of a scientific society (as a student), reviewed 96 journal articles to date, taught nine university courses, and organized conferences and workshops. The result? One phone interview for a tenure-track position out of 17 applications.

I was disappointed of course, but not surprised. It was statistically unlikely that I would land a job coming from an R2 state school program, and even more unlikely given self-imposed geographical constraints. The data are against me, and that’s okay, but the state of the job market is, in my view, ridiculous. And now that people are analyzing job market data, the ridiculousness of the job market is becoming concrete rather than hear-say and anecdotes.

Below, are some job market data resources with some highlights I found particularly telling.


The 2018-2019 SPSP Job Market Survey

This is a phenomenal resource, and very telling about the state of the market. Huge kudos to Heidi A. Vuletich, Fernanda C. Andrade, Diego Guevara Beltran, and Hasagani Tissera for publishing these data.

The main take away from this survey is what does – and does not – predict success on the market. Overall, two primary predictors of tenure-track success emerged: Number of applications and journal impact factors. That’s it. The lottery perspective of the market has been confirmed, and it is still normal for search committees to judge a candidate’s merit on the number of JPSPs they have.

It’s also interesting to see how many publications candidates have. If you chat with professors that were on the market 20 years ago, you know it was unusual to have publications when on the market. Now? The average successful candidate at a research institution has 9 publications. Those at a teaching institution, an average of 6 publications. But, importantly, the number of publications did not differentiate those who did vs. did not land a job. Impact factors are still king.

That very few metrics of productivity predict psychology job market success coincides with other research in the life sciences that concludes, “Traditional benchmarks of a positive research track record above a certain threshold of qualifications were unable to completely differentiate applicants with and without offers.”


The 2017-2018 PsychJobSearchWIki analysis by Mark Thornton

This analysis is mostly descriptive, but nicely demonstrates how limited the market is. How many jobs are there? According to the jobwiki, which seems to be the most representative place on the internet for psychology faculty job postings, there were about 700 in 2017. Cool! Lots of jobs! But when you consider that each area has around 100 or fewer jobs, combined with the fact that thousands of PhDs in psychology are awarded each year, the limited nature of the market is readily apparent.

I am eagerly awaiting an analysis of the 2019-2020 market, which is likely to include new insights into the now standard diversity, equity, and inclusion statements for academic positions.


The 2015-2017 APA Job Advertisement Analysis

A bit older, but still useful, is the APA analysis of job advertisements. The key take-aways here are that there are still not many jobs, with clinical psychology having the highest showing. Many sub-areas of psychology, account for single-digit percentages of the psychology job market.

Also interesting is the distribution of jobs, which unsurprisingly follows the general population distribution in the US. Unless you live the New England area, it is likely that you’ll be moving if you plan to secure a job.


So, if you didn’t land a job this year, even if you did everything ‘right’, it’s more likely a result of the hyper-competitive market and not your skills as a researcher of educator. After weeding out incomplete applications and poor research fit, it is likely the case that there are 20 or 50 qualified applicants, and the decision of who to hire comes down to culture fit and committee preferences. Don’t let a fucked job market diminish your identity as a scholar and academic.

Does Spanking Cause Negative Developmental Outcomes? Yes, But Our New Research Suggests That Effects Are Probably Smaller Than What We Think.

The debate over whether it is moral to spank children is well-regarded as ‘over’ in that there is a strong consensus both publicly and scientifically that it is immoral to spank children. Spanking, defined here as hitting a child on their buttocks or extremities using an open-hand with the intention of modifying undesirable child behavior, is banned in a large number of developed countries with global movements dedicated to banning the behavior. Journalists covering the research on spanking and its effects on child development are equally strong in their conclusions with a number of publications stating that yes, “The spanking debate is over.”

What is behind this consensus view? Mountains of research across family studies, developmental psychology, and social psychology that have culminated on the following conclusions:

  1. Spanking is not associated with any positive outcome.
  2. Spanking does not correct problem behavior. In other words, spanking is not effective.
  3. Spanking is associated with only negative outcomes in children.

The most recent contribution to the spanking literature is an excellent and comprehensive meta-analysis published in 2016 by Elizabeth Gershoff from the University of Texas at Austin, and Andrew Grogan-Kaylor from the University of Michigan. Despite the consensus views regarding the negative outcomes for children as a result of spanking, their meta-analysis addresses some remaining concerns in the literature.

One primary concern addressed in the 2016 meta-analysis is ensuring that spanking (as defined above) is not confounded with harsher forms of physical punishment perpetrated against children (e.g., hitting children with objects; abusive behaviors including choking and beating, in other words, what would be considered physical abuse). This distinction is important: whereas there is a clear consensus that child physical abuse can be detrimental for child development, it is less clear – and still debated – whether controlled spanking is: (1) an effective form of positive punishment that reduces problem behavior, and (2) directly (and causally)associated with negative psychosocial outcomes throughout development and into adulthood.

The results of the Gershoff and Grogan-Kaylor (2016) meta-analysis suggest that spanking is, indeed, associated with wide-ranging negative psychological and behavioral outcomes across development. Psychosocial outcomes resulting from spanking vs. non-spanking, importantly, did not differ as a function of study design (e.g., retrospective, longitudinal), measurement (e.g., observational, child-report, parent-report), or age of children at the time of spanking (ranging from toddlerhood to adolescence). In summary, the results of Gershoff and Grogan-Kaylor align with the general consensus that spanking is consistently associated with poor developmental outcomes, appears to be largely ineffective at reducing problem behavior, and has not been shown to have any positive effects on child development. (Articles covering this paper can be found here and here.)

Although research does clearly show that spanking is associated with negative psychosocial outcomes, not all confounds have been adequately addressed. This limitation of the literature is where my coauthors Eric Connolly, Madi Sogge, Todd Shackelford, Brian Boutwell, and I decided to focus our empirical efforts. The confound we were interested in was genetic confounds. Genetic confounds in this context would reflect the idea that being spanked and the psychosocial outcomes of interests such as externalizing behavior and self-regulation, for example, covary at the genetic level; meaning that the genetic variation underlying children’s propensity to be spanked are the same, to some extent, as the genetic variation underlying their problem behaviors presumed to be an outcome of spanking.

Some research addressing these genetic confounds have been conducted with results generally supporting the notion that genetic covariation can account for a large proportion of the association between spanking and psychosocial outcomes. A study by Sara Jaffee et al. in 2004 found that genetic covariation between spanking and antisocial behavior accounted for 86% of the observed association, whereas the remainder of the effect was primarily explained by nonshared environmental covariation. Another twin study conducted by Button and colleagues in 2008 found that genetic covariation between parental punitive punishment and children’s externalizing behaviors explained between 61% and 98% of the association. (Importantly this is NOT the case for the association between physical abuse and psychosocial outcomes).

Genetically informed research is hard to come by in the spanking literature. Whereas the non-genetic literature includes a wide range of psychosocial outcomes – 17 in the Gershoff and Grogan-Kaylor (2016) meta-analysis – the available genetically-informative research on spanking includes only a limited range of outcomes, such as internalizing and externalizing problems, and antisocial or conduct behavioral problems. The purpose of our research was to use all the information available in the spanking literature to provide an up-to-date analysis of the role that genetic confounds may play in a wide range of psychosocial outcomes. Specifically, we provided probable ranges of estimates of the degree to which genetic and nonshared environmental covariation could explain the reported associations from the Gershoff and Gorgan-Kaylor meta-analysis.

We conducted two studies to address our aims. The first study reported results from the Children of the National Longitudinal Survey of Youth (CNLSY). Similar to the results of earlier studies, our analyses showed that for alcohol use, delinquency, and depression, 49%, 66%, and 80% of the association, respectively between spanking and the outcomes was accounted for by genetic covariation between the traits, with the remainder explained by non-shared environmental covariation.

Our second study took a simulation modeling approach to estimate the extent to which the association between spanking and 17 psychosocial outcomes was likely to be explained by genetic covariation and non-shared environmental covariation. For all outcomes, with the exception of physical abuse, the estimates showed that genetic covariation could potentially account for the majority of the association between spanking and the psychosocial outcome of interest. Non-shared environmental covariance accounted for a relatively small, but non-trivial amount of the association between spanking and psychosocial outcomes.

So, what do these results mean for our understanding of how spanking effects children psychologically? First, our results are consistent with the notion that it is plausible to suggest that spanking, in some cases, may exert a negative impact, and certainly not a positive impact, on psychosocial development across the life course. However, there is an important caveat to this claim: The effect sizes are likely to be much smaller than what the current literature suggests. Overall, the results of genetically informative research suggesting moderate-to-large degrees of genetic covariation between spanking and psychosocial outcomes imply that once genetic variation is considered, the environmentally-mediated (casual) effects of spanking on psychosocial outcomes will be much smaller in magnitude than what non-genetically-informed studies suggest.

That the true effects of spanking on child development are likely smaller is magnitude than what the reported literature suggests is not at all to say that the negative effects of spanking on child development are non-existent. The extant research and the data in our new study strongly suggest that non-shared environmental covariation explains a non-trivial proportion of the phenotypic effect – a finding that is, in fact, consistent with causal interpretations of the negative effects of spanking.

Although the spanking debate may be ‘over’ with regard to whether spanking negatively effects children’s development, our understanding of the effects from a genetically informative perspective is far from resolved. Our work demonstrates how little genetically informative research has actually been done on the topic, and the potential impact that genetically informative research can have on the estimates we rely on to make informed policy decisions and moral arguments. So, yes, all research points to spanking having negative effects on psychosocial outcomes, but the magnitude of the effect is likely to be much smaller thank what we currently think to be the case.

You can find the OSF project here, and the pre-print of our paper here, which is currently under review for publication.

Photo by Caleb Woods on Unsplash

Thoughts on Human Diversity

Human diversity is complex and controversial, but we know more about our diversity now than at any other point in human history. Charles Murray’s latest book, Human Diversity: The Biology of Gender, Race, and Class, has stirred up op-eds and disagreements in which critics have correctly focused on Murray’s suboptimal description about polygenic scores and the presumed causally between genetic associations with phenotypic traits. The points critics have raised are important and although I largely agree with these criticisms, others have already sufficiently discussed them. Here, instead, I want to focus on my thoughts about the race section of Human Diversity, and Murray’s argument against the idea that race is a social construct.

I have remained intentionally mute on the ongoing scholarly debate of race differences in psychological traits. The debate can be summarized as two oversimplified positions. On one side of the debate, the argument is that race is a social construct in which proponents assert that race differences in skin color or other outwardly visible traits do not correspond to meaningful differences in psychological traits, with IQ being the primary trait of focus. The other side of the debate argues that race is not a social construct, and there are real, partly genetically based differences between races in psychological traits, including IQ.

This debate about the ‘realness’ or ‘biologicalness’ of race intersects with various orthogonal questions: Can the term “race” be used interchangeably with “population”? What evolutionary processes are responsible for any observed differences? If natural selection is responsible for these observed differences, what are the selection pressures that prompted the differences?

Murray, unsurprising to many reading this I would presume, argues against the ‘race is a social construct’ position and states that there is sufficient evidence to reject the proposition that race is a social construct. Human Diversity includes three chapters dedicated to laying out his argument for why race is biologically real. The first chapter focuses on genetic distinctiveness among ancestral populations; the second on evolution since humans left Africa; and the third on population differences in genetic variants associated with psychological traits. This evidence, Murray argues, is supportive of the position that race is a biologically ‘real’ entity.

I disagree.

For a few years now, I have been passively observing the biological race debate unfolding between academics and intellectuals online, wrestling with the arguments and data to come to a confident conclusion about my own position on this topic. Reading Human Diversity has led to the culmination of my position that, yes, race – as has been historically and culturally defined, predominantly by skin color – is more of a social construct than a biological reality. Independent of this conclusion, even if one were to hold the view that race is ‘biological’ Murray’s argument for such a view is simply unconvincing.

The basis for my position that race is more accurately described as a social construct rather than a biological reality rests on two primary points, described in detail below. First, using the terms “race” and “population” interchangeably is inappropriate and inaccurate.  And, second, the extraordinary genetic diversity within continental regions is largely ignored, and currently, too limited to draw convincing conclusions about race differences in psychological traits.

Importantly, however, I do not disagree with the data that Murray presented in these chapters, but I do disagree with the conclusions he draws from them, and I find his presentation of the argument for the position that race is not a social construct inappropriate given the data he uses as support.

I do agree with the claim that human populations differ genetically and for many evolutionary-related reasons. This is an indisputable fact of modern genetic findings that cannot be dismissed. But that human populations differ genetically is not synonymous with the claim that human races differ genetically. Using “population” in place of “race” is a slight I’ve seen used to support the position that races – Black, White, Asian – differ genetically in meaningful ways that contribute to psychology and behavior, which I view as incorrect.

This inappropriate term-switching is used by Murray often in the race chapters of his book. The inconsistency is apparent by looking at the title of the book and comparing it with the chapter titles and propositions in the race section of Human Diversity. To argue against the position that race is a social construct, Murray inappropriately uses data about human populations. Although he defends his term switching by rightfully stating that race has “outlived its usefulness when discussing genetics” (p.135) he then proceeds to inappropriately use the term race throughout the chapters, and uses populations data to support his claim that race is biologically real. (At one point he even refers to other hominid species as “races” of humans.)

That human populations differ genetically, and that Africans, Europeans, and Asians have some broad-level statistical covariation in models is a more appropriately a reflection of human migration patterns over evolutionary history, than meaningful categorical distinctions for understanding psychology. As the number of groups in these statistical models increases, populations begin to split off by regions within these broad continental groups, as Murray shows. Specifying three, seven, or 15 groups, however, is a rather arbitrary decision. Imagine you are an alien visitor with only the genetic data that Murray describes in Chapter 7. Would you pick three groups? Seven? Why? Without pre-existing cultural concepts of race, it is hard to justify an answer that corresponds with socially derived racial groups.

Another problem with these kinds of data is that the data we have from populations within Africa and Asia are incredibly limited relative to that of populations from European. The genetic diversity in African populations is notoriously diverse, yet we have comparably minimal data. As Dr. Adam Rutherford discussed in his book, two Africans are likely to be more genetically distinct from each other than either is to a European. To simply lump all Africans into a single racial group is lazy, misinformed, and more of a reflection of the dearth of data that we have from the continent – an argument that Rutherford articulates elegantly in his book, A Brief History of Everyone Who Ever Lived.

The lack-of-data problem leads me to another criticism of Murray’s argument, which he outlines in Chapter 8: that “evolutionary selection pressure since humans left Africa has been extensive and mostly local”. I have two problems with this line of argumentation. First, what seems to be ignored here is that evolution didn’t stop in African populations when some humans migrated out of Africa – populations in Africa continued to evolve and change, too(!).

The chapter describes evolutionary processes responsible for changes in genetic frequencies between populations, and how migration causes reduction in genetic diversity in the migrating populations. But, the ability of genes to vary (for various evolutionary reasons) across populations does not necessitate that there is, or has been, selection pressures to build adaptations between populations, nor does it make it necessarily more likely. The fact that migrating populations can contribute to genetic differences between Africans (as a group) and Europeans (as a group), must also apply to different (migrating) African populations. The argument is not specific to races; though when Murray frames his argument as ‘evolution post-Africa is extensive’, the evolutionary changes within Africa are ignored.

The second problem I have with this argument is that nearly all the evidence for local adaptations across populations are related to specific physiological process or medical disease traits, not psychological traits. Murray and most other’s arguing for his position describe several population specific traits: sickle cell, altitude adaptations, lactose tolerance, etc. (which don’t even vary consistently across racial lines anyway). The primary difference between these kinds of adaptations and the proposed psychological traits that proponents of Murray’s position argue for, such as IQ or personality, is that the physiological and disease traits have clearly defined selection pressures underlying their evolution and extensive empirical support. Proposals for psychological race differences, however, do not, nor is there strong evidence in support of the claims that I have read. A recent study by doctoral student Kevin Bird tested this idea specifically and shows that IQ differences between Blacks and Whites are not genetically driven nor is there evidence of natural selection operating to produce differences.

Although I think evidence of race differences in psychological traits is unlikely to be (accurately) found in the current research environment, I do not think it is unreasonable to hypothesize and explore questions related to psychological differences between populations (or races). However, it seems to me that the likelihood of finding differences between races is more unlikely than finding differences between populations. Because the variation within races (e.g., ‘blacks’ and ‘whites’ as broad groups) is undoubtedly larger than the variation within specific human populations, the effect sizes for ‘race’ differences would have to be large relative to population differences to be accurately detectable.

Moreover, as I mentioned above, we have an extremely limited amount of genetic data from African populations, relative to populations in Europe and (east) Asia. Thus, current results comparing data from one European population and data from one African population, for example, tells us only about those two specific populations, not about Whites and Blacks as racial groups. More accurate information will be known as the amount of genetic data from non-European populations becomes comparable to that of European populations, and problems of comparing across human populations is resolved.


Murray’s claim that we should move on from the use of race in genetics in favor of population is one I agree with. But that is not consistent with the claim that race is not a social construct. Race, as he describes in Chapter 6, came into the cultural lexicon to categorize people by different skin color and other outward features. What genetics has done is to clearly show that race as skin color or outward features is not a proxy for genetic variation of human populations. Two populations can look the same, but be genetically distant. As Rutherford explains in his book, using ‘black’ skin as a categorical marker hardly makes sense.

What genetics has done for our understanding of race is analogous to what happened in taxonomy with the evolution and genetic revolutions. As described by Dr. Carol Yoon in Naming Nature, taxonomy historically was based on the outward appearance of organisms. Humans have all sorts of culturally useful names for categories of animals, such as fish, birds, and bugs. These categories of animals, like humans, contain enormous diversity within them. Dolphins and trout both swim, bats and magpies both fly, but these outward characteristics from which we group them do not correspond to the biological reality of their relatedness.

This is what has happened with race, in my opinion. As our knowledge of genetics and human diversity has flourished, race – once a seemingly reliable, accurate, and obvious way to categorize human groups – has become a social construct with the acquisition of knowledge. We now know that social categories of race as defined by skin color are not a reliable and accurate way to categorize human groups. So, Murray is right to say that race is no longer an appropriate way to discuss human diversity. He is wrong to conclude from this that race is therefore not a social construct. Race, in my view, is as biologically real as the socially derived, yet meaningful, category “fish”.

Are Replication Rates in Psychology Actually Better Than Expected? (Despite Them Still Being Terrible, Of Course)

I’ve been doing research into open science and the replication crisis in psychology recently for work (to make a case for open science practices in the education technology industry), and stumbled across this article on the replication crisis in psychology by Ulrich Schimmack. One particular part of the article caught my attention (highlighted in the image below).

The idea here, as I interpret it, is that if psychology studies are conducted at 80% power on average, then the replication rate in psychology should be around 80% — assuming everything is done correctly. The replication rate in psychology, however, is very clearly not at 80%. A quick summary of the major replication projects in psychology, such as the Open Science Collaboration and the Many Labs efforts, point to a replication rate closer to around 46%.

So, assuming standard study power of 80% we would then expect that 20% of studies wouldn’t replicate even if everything was done correctly in the replication studies. But over 50% of psychology studies fail to replicate and psychology studies aren’t powered at 80% — not even close. This comment in Ulrich’s article reminded me of a 2018 study published in Psychological Bulletin on the average power in psychology studies across sub-disciplines. The result? The average power in psychology studies is a whopping 36% (just shy of 80%, yea?), with social psychology – the main target of replication efforts – at just over 30%. Only around 20% of social psychology studies are adequately powered, according to this study.

What does this mean, then? Well, if the average power of psychology studies is only 36% then we would expect that on average about 64% of psychology studies wouldn’t replicate which is pretty close to the estimated 54% failed replications across the major replication effort projects. So, in a way the “shockingly” low replication rates are completely expected given the dismal power of psychology studies, on average.

But there is reason to suspect that the replication rate of around 46% is actually pretty darn high, all things considered. I was curious, so I conducted rigorous Twitter polls to see what the masses thought. Below, are the two polls. The first, was trying to get at what my biased twitter audience thought the replication rate would be assuming that all psychology studies have on average 80% power. The majority thought replication would be closest to around 50% — close the current rate. The second poll simply asked if the statement by Ulrich (highlighted in the opening paragraph of this post) was true or false. A close poll indicated that it was ‘true’, but many thought it was false.

Finally, once all the other methodological issues are considered (e.g., variation in the replication study, poor measurement, inaccurate effect sizes, etc), the expected replication rate – even with 80% average power, would likely be lower than 80%. All of this therefore leads me to the conclusion that given the dismal average power of psychology studies, in addition to other methodological problems (e.g., measurement problems), and research process biases (e.g., p-hacking, HARKing), the psychology replication rate of around 46% is way higher than what would be expected. Does this mean that it is good? Absolutely not.

Thoughts?

Brief Book Review: The Education of an Idealist


Memoirs are a new genre I’ve been exploring recently, The Education of an Idealist being my third over the past few months, preceded by Educated: A Memoir and Becoming (Michelle Obama needs no subtitle). The common thread throughout each has been the personal education journey of the subject, despite the actual journeys of Tara, Michelle, and Samantha being exceptionally different.

Samantha’s journey began in Ireland and followed her to America when she was a young girl. She ferociously absorbed American culture throughout her childhood becoming intensely passionate about baseball – a sport which I could never quite grasp the interest of. The aspect of her story which I liked most was that her path wandered, beginning in sports journalism, and ending as the United States Ambassador at the United Nations. The stories of many successful individuals, in contrast, tend to look linear in hindsight: a determined person with a clear goal since adolescence. A story that is true of only a select few.

Samantha was humble throughout her story, intimately sharing her deepest insecurities, uncertainties, and fears with the reader. She acknowledged when serendipitous events, like meeting Barack Obama, were pivotal moments in her life course, as they so often are. But it is nonetheless comforting to hear about someone who reaches the echelon of their career by following a meandering path sprinkled with gracious opportunity.

The part of her story that was most interesting to me was her role as a foreign correspondent in the Balkans during the Bosnian genocide in the 1990s. There she learned of suffering, geopolitics, risk, and reporting. I had never before read from the perspective of a foreign correspondent, especially not during a tumultuous time in Balkan history. It was a particularly engrossing part of the book, perhaps because of the salience of being in a war zone and perhaps because I was too young at the time of the events to know of their occurrence.


But it is nonetheless comforting to hear about someone who reaches the echelon of their career by following a meandering path sprinkled with gracious opportunity.


Because of Samantha’s roles – as a foreign correspondent, as an aid for then Senator Obama, and finally as UN ambassador – her book not only served as story worthy of sharing, but also as an interesting history and political lesson. A substantial portion of her book focused on Balkan history, but an equally substantial portion of her book focused on UN international relations, particularly with Russia. She plainly portrayed the difficult position the United States is in when human rights issues intersect with geopolitical ones, such as the recognition of genocide on the world stage, and action in response to the use of chemical weapons in Syria.

Samantha’s story showcases the contrasting reality of geopolitics with idealistic goals – goals from which she admirably never wavered on. Her educational journey throughout her story is evident – in her career and personal life. Learning about her story, and those of Tara Westover and Michelle Obama, demonstrates how deeply personal education really is, the true essence of which is too often lost in current educational institutions.

Brief Book Review: Mind in Motion


A goal of mine for 2020 is to write more — a goal many academics commit to each year. This year, however, as I have stepped away from the ivory tower, I am going to make a concerted effort to write more public essays and blog posts now that I am far less tied to the necessity of formal academic publishing. As part of this goal, I am introducing a new series, Brief Book Reviews, where I will do exactly as the title suggests: I’ll offer brief reviews of (hopefully) all the books I read this year. I hope to provide readers with at least a couple of interesting recommendations. (To see my 2019 book recommendations, see here)

First up for 2020 was Mind in Motion: How Action Shapes Thought, by Barbara Tversky. I first heard of this book on Sam Harris’ podcast where he discussed Tversky’s interesting thesis that action and motion are the basis of abstract though. The thesis was phenomenally interesting as, evolutionarily speaking, we (and other animals) evolved diverse means of communication previous to the evolution of our remarkable and complex language systems. Her thesis is doubly interesting in that one hypothesis for the ultimate function of the brain (or, centralized nervous system) that I find enchanting is that the brain ultimately evolved to facilitate locomotion — all animals must move; and, those that do not, typically don’t need a brain. Take, as a favorite example of mine, the sea squirt (and other related species) which, upon locomoting to a suitable permanent location to attach to, it promptly digests its own brain — it is no longer needed.

To continue reading, click here.

This post originally appeared on my Medium account on 22 January 2020.