Personal Data Science: What, Why, and How

Introduction

A leader of transformation, and managers involved,
need to learn the psychology of individuals,
the psychology of a group,
the psychology of society,
and the psychology of change.

Some understanding of variation [i.e. statistics],
including appreciation of a stable system,
and some understanding of special causes
and common causes of variation,
are essential for management of a system,
including management of people.
—William Edward Deming
The New Economics for Industry, Government, Education,
2nd Edition, p. 95.

You’re the only one who can manage yourself, and here’s a key point.
No mood expert I know of would ever talk about curing depression.
Keep this point from this lecture.
The goal is not to cure depression.
The goal is to learn how to manage your moods.
These are things you have to manage every day of your life,
and managing your moods requires the skills—
psychologists call it self-regulation,
but the simple phrase managing your mood counts.
—Michael D. Yapko, Ph.D, Australian Psychological Society
How to recover from depression

Dear Young Tim,

Congratulations on making it to the end of 2021! Extra congratulations for making it here without any permanent debilitations, mentally or physically. I wasn’t sure we would see the end of the year in this state, so there is much to celebrate!

Despite this good news, I’d like to offer you an apology. Looking from hindsight, I did not protect us and meet our needs. Accordingly, you and your twin protector of all-encompassing-procrastination depression are fully active. I’ve left you two running about, trying desperately to hold onto ourselves and the few remaining goals we’ve carried to the end of the year.

In all honesty, relying only on our mind’s awareness and on our rotating present attention to keep track of things, has left us exhausted and in shambles. It has left us in a severe local minima. Young Tim, in our hero’s journey, we’re at the point of having begun the journey, having achieved some success, but nonetheless having come to hard times. Now, it’s time to change in order to experience better times in the future.

Part of this change will involve seeing therapists and coaches. They’re absolutely helpful. However, we nevertheless face the decision of “how will we interact with mental health professionals?” As statistician Talithia Williams so aptly stated,

Medical doctors [and mental health workers] are experts in the population,
but you are the expert on yourself.
And so when two of you come together,
when two experts come together,
the two of you are able to make a better decision
than just your doctor alone.

In a real sense, we need to be head of “personal data science” for our own mental health management. There are good economic reasons for this. For one, I have the greatest competitive advantage for this role. I have direct access to what is going on in my mind at any time. I can perform the data collection whereas others cannot. I am also a professional data scientist. Who is better suited to analyze my data productively, quickly, and thoroughly?

Finally, this data science for mental health role has a huge gap. The role is not filled comprehensively or well by the mental health professionals I have interacted with. Put another way, no one in my life, neither I nor anyone else, has been using knowledge of how my personal mood systems vary in combination with principles from psychology for the management of my moods. I suspect that I can play this role better than others, and since it is not being done by others, I may benefit from doing it.

What is personal data science?

This discussion brings up the question of what is “personal data science,” before we go too far down the rabbit hole of “why may I want to do this?” The basic idea is that personal data science is the application of data science perspectives and techniques to the betterment of personal problems that one is facing, with the aim of leading or managing oneself more effectively. Note this includes using thick qualitative descriptions and story elicitation, as well as using quantitative data collection and analysis. See the image below for a full overview.

As mentioned earlier, one immediately relevant personal problem is the management of our emotions and moods on a daily basis such that we accomplish our goals. Another related personal problem involves ensuring that our exercise and nutrition maintain our good health over the long-term. Finally—but not comprehensively—we might ask how can we use physical health outcome data such as our temperature, our blood pressure, our blood glucose levels, and our heart rate? If we collect and store our observations—our quantitive stories— about these important topics, then we may be able to use qualitative and quantitative analysis methods on the collected data.

Why should we maintain personal data science projects?

Now that we know what personal data science is, why should we spend our limited and precious time doing it? As mentioned in the introduction, we have a competitive advantage in this subject. However, the case for practicing personal data science is deeper than this:

Young Tim, “as knowledge workers, our mood is one of the largest determinants of our productivity” (Tiago Forte). This is true whether we are doing work to better our own lives or for others. Each of us is affected by our own moods more frequently than any other person. Therefore, each of us bears the highest responsibility for managing our own moods.

As quoted by W. Edwards Demings—a pioneer of modern American management—, management requires knowledge of both psychology and of variation (i.e., statistics). Essentially, everyone needs to be their own data scientist, with subject matter expertise in psychology.

Of course, if we take up this role, benefits will come our way. We’ll gain increased self-knowledge by paying more consistent attention to ourselves; we’ll be able to exercise greater self-control through our increased self-knowledge; and we’ll be able to achieve greater quality of our own lives in the future— we’ll be better able to thrive on our own as demand for therapists far outpaces the supply of therapists (both in absolute number and in quality hours).

In addition to immediate and future benefits to our own lives, we can help enable future benefits for other people. If we are able to execute useful personal data science projects then we may be able to make helpful tools and distill useful stories for our families, our social circles, and the public.

Finally, on a personal and practical level, if we make everything a data science project, then we gain

  • immediate calm because these projects are familiar to us;
  • immediate clarity, because we have reasonable, default project starting points;
  • immediately greater courage from our history of long-term data science success despite adversity;
  • future analytical benefit from accurately storing and retrieving data as opposed to inaccurately reconstructing memories; and
  • future analytical benefit from collecting pieces of externalized thought that can be more easily manipulated than they could be in our working memory.

How to get started?

Given these benefits, the immediate question is–how can we get started? I believe the thing to do is to leverage Susan Collins observation that “emotions are data.” As far as personal data science goes, a critical data science project to begin with is collecting data on and understanding our emotions and moods. The reason for this is that if we feel bad, then our attempts to solve any other problem become more difficult.

To get started, focus on infrastructure basics. Specifically, consider the “golden path” shown in the diagram below.

Start by creating a digital folder. Let this folder house all of our data science projects, and yes, there will be multiple.

A helpful structure for our overall folder could be the P.A.R.A. folder system. That is, four sub-folders: projects, areas, resources, and archives. Each of our data science projects will live in its own repository within one of these four sub-folders.

Next, in areas/, make a repository for the data science project of managing one’s moods and emotions. In this repository, we’re going to track how we’re feeling throughout the day and across days. Some guidance on how to perform emotion or mood tracking is described here. See also the moodmeter app and process.

An easy way to set up the repository is to use the cookiecutter folder template that I set up for data science projects. We’ll also want to initialize software for version controlling all of our files, such as git for code or files we may wish to frequently change and git-annex for media files. This will get us to a state where we have a local “home” on our computer for our work.

On a mental level, our first goal will be to understand how our moods fluctuate, and in relation to what circumstances. Our second goal will be to use this knowledge to regulate the moods and emotions we experience, aiming to increase the presence of “Self-emotions”: the 8C’s and 5P’s. My rationale is that doing so maximizes our chance of achieving our goals. In particular, we’ll want to leverage our emotions to support our health in all the ways we can: physically, mentally, emotionally, socially, romantically, and financially.

To do so, after we’ve made an emotion or moods repository, make a note listing all the problems that we can think of: problems that either stem from or cause our emotional dysregulation. This will overview the problems that we are dealing with— i.e., the full-set of problems we’ll be transforming and that data science might help us reduce.

After the setup described so far, we’ll have a local but insecure folder for all of our personal data science work. We’ll want to take additional steps to secure our data both on our computer locally as well as on external hardware and over the internet. The full setup we’ll use is illustrated in the golden path diagram above, but we’ll describe it in-depth in future posts.

Young Tim, if you make this turn towards organizing everything in your life as personal data science projects, then I think you’ll be able to put yourself back in order again. And what a fun ride it’ll be. Fear not, for you’re well prepared for this role: more prepared than for any other thing we’ve done to date, and I’m excited to see what good it amounts to.

Till next time,

P.S.: For others’ excellent takes on why personal science and what we can gain, see the following peer reviewed publications: