On good models and good research

Dear Young Tim,

If I know you, and I’m pretty sure I do, then you will want to maximize the amount of “great work” that you produce in your life.

You have read all about Richard Hamming’s (1986) calls to “drop modesty and say to yourself ‘[y]es, I would like to do first class work’.” You have read Cal Newport’s (2012) treatise on how a good life is made more probable by doing work that is “so good, they can’t ignore you.”

By now, you have bought into the idea, and you are ready to try your hand at doing the best scholarly / scientific work you can. Wonderful. The problem, however, is that you have not done any great work yet. You don’t know how to do it, and you’ll be figuring it out for a long time (if not forever). One approach is to simply start working, and then iteratively try to produce better and better work.

For sure, that is one way to go, and it may work… eventually.

A better approach, I believe, is to first create a mental map for yourself—a characterization of what good work actually looks like. Then, once you have a clear picture of your target, you should be able to achieve your goal more quickly. You will make more consistent progress if you can recognize your goal and keep it in sight; you will better able to recognize when you are going astray, thereby wasting precious time; and you will be better able to know when you actually produced good work and should take a well-earned break to celebrate.

Of course, it helps if I can be specific. Your work—your research—involves building statistical models to describe, predict, and hopefully influence reality. Without further ado, here is my best attempt at painting a picture for you of what good research and good models look like.

Good research

Let’s start broad, with research as a whole, and then we’ll narrow our focus to modeling later.

At the highest level, I think good research should do things. In fact, I believe good research should do many things, all at once. And as we both know, multi-tasking is hard! In this case, I think it’s hard, but worthwhile…

Here then are the things I think good research should do, along with links to the many people/places that seeded these ideas in the first place.

Overall, I believe good research should

  • expose one’s ideas for the express purpose of scrutiny and criticism. This is how truth and progress are discovered and how society inches closer to them.
  • demonstrate why and how past ideas were wrong or incomplete.
  • answer a fundamental / important question (i.e., rule 8).
  • stand on the shoulders of giants (i.e., be thoroughly rooted in prior work).
  • tailor its methods to the questions being asked (i.e., qualitative and quantitative methods, in their many varieties, all have their uses and time to shine).
  • be based on adequate data for the question at hand. Note I mean adequate both in size and quality (i.e., measurement is important).
  • be rigorous. All techniques used should be thoroughly interrogated for appropriateness / external validity / internal validity.

In additional to the points above, if we go further to think of methodological research (in the sense of statistical methodology), then I think one’s research should:

  • report substantively significant, not just statistically significant, results.
  • provide artifacts (e.g., software, procedures) that are practically useful outside of academia.
  • paint a “thick” picture of a method:
    • Why does the method work?
    • When should the method not be expected to work well?
    • When should the method be expected to work well?
    • How robust is the method when its justifying conditions are not met?
    • What indicators tell one that the method is working?
    • What indicators tell one that the method is not working?
  • understand when to use descriptive versus predictive versus causal inference methods. (And also understand when such distinctions are blurred or unnecessary.)
  • be (even more) rigourous, i.e., make use of:
    • code/software tests (e.g., unit tests)
    • numeric tests of the validity of one’s methods (e.g., goodness-of-fit tests)
    • graphical tests of the validity of one’s methods (e.g., posterior predictive checks)
  • recognize uncertainty as a first-class citizen. At the end of the day, quantifying and reducing uncertainty is the main point of one’s statistical methods.

Good models

Alright, I understand you might be feeling overwhelmed (“unit tests; posterior predictive checks—what are those?”), and you may want to go back to whatever you were doing before you started dreaming about your future self. But before you do, let’s talk about good models. Given the extent to which you will find yourself building models in the future, I think this topic demands an entire section to itself.

On the whole, I think good models should:

  • be reproducible and replicable, (i.e., release the source code and data (or a representative dataset if privacy concerns are an issue)).
  • be thoroughly checked / falsified / investigated
  • provide a great description of reality / the data it is based on (i.e., pass the many checks described above).
  • be the result of an iterative process of model-building and model-criticism (i.e., Box’s Loop)
  • be only as complicated as necessary to adequately describe reality. What is adequate is of course up for debate (and debate you will).
  • be tailored to the problem at hand. Cookbook applications of existing methods will rarely be completely satisfying.
  • explicitly represent the treatment assignment mechanism in addition to the outcome, if one’s goal is to make causal inferences from the model.
  • be well understood. We should understand why and when our models work and fail.
  • be extensively visualized. We should never trust any summary statistics on their own, and this includes parameter estimates. Statistics may lie, but looking at the data and models reveals many truths.

How to proceed

Together, all of the bullet-points above may seem like a lot. It is.

You will likely not meet all of these criteria in all of your research projects. In fact, none of your model building efforts from your PhD will meet all of these criteria. That is fine! Remember, we all start somewhere.

The point is not to beat yourself up about it, but to learn as quickly as possible and always do better the next time.

Of course, while following this path, you will occasionally find yourself in disagreement with others over how to produce good work. At those times, you should first try to learn from the other viewpoint. Extract all the good that you can, and do not presume that you know best or that the other viewpoint is wrong. Second, once you have learned all you can, if the opposing viewpoints still appear problematic, you should try improving the research aesthetics of the environment within which you are operating. Try hard, as most people around you will also want to do great work.

However, if you are unsuccessful in creating an environment that is conducive to you doing work that you believe is good, you should leave the situation as quickly as possible! In places where your research aesthetics do not align with one’s larger group or organization, you will not be appreciated to the extent you desire. Your internal reputation will run the risk of suffering due to misalignment, and your external reputation will be at risk if you are associated with work that does not fit the description of good to which you ascribe. As in the 2017 film by Jordan Peele, Get Out!

In all seriousness though, heed the words of Jan Fields (2011) to

Remember that your reputation is everything. You build your personal brand through everything you do, whether big actions or small decisions, and that brand will stay with you throughout your career.

Build a career that you can be proud of by always doing the best work you can.