Thursday, December 19, 2013

A discussion of causal inference on fMRI data

My facebook page was recently home to an interesting discussion about causal inference that was spurred by a recent article that I retweeted:

Effective connectivity or just plumbing? Granger Causality estimates highly reliable maps of venous drainage.

What follows is the discussion that ensued.  I have moved it to my blog so that additional people can participate, and so that the discussion will be openly accessible (after removing a few less relevant comments).  Please feel free to comment with additional thoughts about the discussion.

  • Peter Bandettini Agree! Latencies due to veins 1 to 4 sec. Latencies due to neurons < 300 ms. Vein latency variation over space dominates.
  • Russ Poldrack it's amazing how we have known for a good while now that Granger causality is useless for fMRI, yes people continue to use it
  • Peter Bandettini There are ways to look through the plumbing.
  • Tal Yarkoni My impression is that as bad as the assumptions you have to make for GCM may be, the assumptions for DCM are still worse. The emerging trend seems to be towards a general skepticism of causal modeling of observational imaging data, not a shift from one causal method to another.
  • Micah Allen In terms of hemodynamics, I'm not aware of any assumptions different from those in all canonical HRF models. DCM uses a slightly updated version of the balloon model. Further, unlike GCM the parameters estimated by DCM have been repeatedly validated using multimodal imaging. Although I will bring this up at the next Methods meeting.
  • Russ Poldrack Tal - I don't agree regarding blanket skepticism about causal inference from observational data. It is definitely limited by some strong assumptions, but under those assumptions it works well, at least on simulated data (as shown e.g. by the recent work from Ramsey and Glymour). But I do agree regarding general skepticism about DCM - not so much about the theory, which I think is mostly reasonable, but about the application to fMRI (especially by people who just punch the button in SPM without really understanding what they are doing). I thought that the Lohmann paper did a good job of laying out many of the problems, and that Karl et al.'s response was pretty much gobbledygook.
  • Vince Calhoun I do not think even GCM can be considered useless for fMRI...however it is critical to keep the assumptions (what are the parameters capturing) and the data (what causes hemodynamic delay, etc) in mind in evaluating its utility and especially in interpreting the results (e.g. does this approach enable us to 'see through' the delay or just to characterize it). The post analysis (single or group) is critical as well and will often float or sink the boat. Every method out there has lots of holes in it, but in the right hands most of them can be informative.
  • Tal Yarkoni Russ, I think the question is whether you think those strong assumptions are ever *realistic*... E.g., looking back at the Ramsey & Glymour sims, my impression is that it's kind of an artificial situation in that (a) the relevant ROIs have already been selected (typically one wouldn't have much basis for knowing that only *these* 11 ROIs are relevant to the causal model!), (b) the focus is on ROI-level dynamics when it's not at all clear that connectivity between ROIs is a good proxy for neuronal dynamics (I understand this is a practical assumption made of necessity, but we don't actually know if it's a reasonable one!), and (c) I think the way error is quantified is misleading--if you tell me that you only have 3 mislabeled edges out of 60, that doesn't sound like much, but consider the implication, which could be, e.g., that now we think ACC directs activity in IPC rather than vice versa--which from a theoretical standpoint could completely change our understanding, and might very well falsify one's initial hypothesis!

    And this still ignores all of the standard criticisms of causal modeling of observational data, which still apply to neuroimaging data--e.g., the missing variable problem, the labeling problem (if you define the boundaries of the nodes differently, you will get very different results), the likelihood that there's heterogeneity in causal relationships across subjects, and so on. So personally I'm very skeptical that once you get beyond toy examples with a very small number of nodes (e.g., visual cortex responds to a stimulus and causes prefrontal changes at some later time), it's possible to say much of anything *positive* about causal dynamics in a complex, brain-wide network. That said, I do agree with Vince that any method can have utility when handled carefully. E.g., a relatively benign use of causal methods may be to determine whether a hypothesized model is completely *incompatible* with the data--which is a safer kind of inference. But that's almost never how these techniques are used in practice...
  • Tyler Davis Pragmatically, if you take all of the nonsense that happens with SEM in the behavioral literature, and then multiply be a factor of two to account for the nonsense due to mismodeling/misinterpreting hemodynamics, it makes it tough to believe an fMRI study with causal modeling unless you know the authors are good at math/know what they are doing or you were inclined to believe the results already.
  • Russ Poldrack We are using a measurement method that intrinsically is at least an order of magnitude from the real casual mechanisms, so It would be folly to have too much faith in the resulting causal inferences. But to the degree that we can validate them externally, the results could still be useful. For me the proof is always in the pudding!
  • Anastasia Christakou What *is* the pudding? If you have a biological hypothesis and your model-driven analysis is in line with it, what more can you do to "prove" it? Are you talking about *independent* external validation?
  • Russ Poldrack the pudding in this case is some kind of external validation - for example, if you can show me that graphical model/DCM/GCM parameters are believably heritable, or that model parameters from fMRI are predictive of results in another domain (e.g., EEG/MEG, behavior).
  • Micah Allen Tal Yarkoni I don't really find the ROI/model-selection problem with DCM overly troubling, although I do see where you are coming from. As Karl is fond of saying, it's your job as the researcher to motivate the model space as it's the hypothesis you are testing. The DCM is only valid in light of that constraint. I find it a bit unfortunate that there is this divergence between the two schools of thought; obviously in cases where you have no clue what the relevant hypothesis should be, tools like graph theory and functional correlations can be an excellent starting point. By definition, the validity of any DCM depends upon the validity of interpreting the experimentally-induced mass-univariate activations. DCM is built to assess specific directional hypothesis between regions activated by some experimental manipulation - it is unsurprising that if the selected task activates brain regions non-specifically or the ROIs are extracted haphazardly then the DCMs are equally invalid. But this isn't an indictment of DCM, it's a rather a failure to motivate a relevant model space. If working in unclear or murky territory, Karl is the first to say that a graph theoretical or connectomic approach can be a first step towards motivating a DCM. This is all circumstantial to the basic issue that GCM just plain gets the direction of connections wrong. 

    I've read quite a bit of the DCM and GCM literature, and I actually do agree with you that many of the early papers are plagued by extremely toy examples. The end result was also always the same, that everything connected to everything. This is part of why the best DCM papers are those based on computational models, such as Hanneke den Ouden's papers on motor learning, where the estimated parameters and winning model families are themselves of interest. DCM is extremely well equipped to assess the evidence that say, motor cortex updates premotor with the probability of a stimulus being a face or a house. This can't be said for any other method - DCM is fundamentally built to assess brain-behavior relationships in this way. All the connectomics in the world won't tell you much beyond "these brain regions are really important hubs and you probably shouldn't knock them out". To go the rest of the way you need a hypothesis driven methodology. 

    That being said, a previous limitation has simply been that it's incredibly frustrating and time consuming to estimate interesting models with more than 2 or 3 nodes. This has resulted in many small sample, 'toy' DCM papers that are generally not very interesting. The new post-hoc model optimization routines largely ameriolate these practical concerns - in my upcoming paper I easily estimate models with all 6-7 nodes activated by my mass-univariate analysis. I have a colleague estimating similar models in 600 scans from the HCP database. This means we will begin to see more intuitive brain-behavior correlations. 

    As for the debate between Lohmann and Friston - Lohmann's critique is just plain factually wrong on the details of the post-hoc procedure, and on several other details. Further Lohmann seems to fundamentally misunderstand the goal of model development and model selection. So I'm not really convinced by that. 

    DCM requires strong hypothesis, which is both it's strength and weakness, and fits extremely well with exploratory data-driven methods that actually work as advertised (unlike GCM). We've not even gotten into DCM for MEG/EEG (on which I am not an expert). The neural mass models there are extremely fascinating, going far beyond modelling mere data features to actually encapsulating a model of the underlying neurophysiological brain dynamics underlying observed responses. DCM for fMRI is itself at best just a starting point for actual neural modelling in M/EEG. 

    Finally, the greatest strength of DCM is the Bayesian model selection procedure. Don't like the canonical HRF? Substitute your own HRF and compare model evidences. Want to combine DTI, MEG, and LFP? DCM can do that.
  • Russ Poldrack Micah - what about Lohmann's comments regarding model fit? I read Karl's response as saying that it doesn't matter if your model only accounts for 0.1% of the variance, as long as it beats another model in the model selection procedure. That seems mathematically correct but scientifically wrongheaded. FWIW We gave up on DCM soon after starting when we found that the models generally fit the data really poorly.
  • Micah Allen Were you doing DCM for fMRI or MEG? In general I've been told that the MEG variance explained tend to be far higher than in the fMRI DCM - close to 90% in some cases, compared to 20-60% in the best case for fMRI. An important caveat with DCM for fMRI is that it is not ideal for fast purely event related designs. If you have a fast stimulus driving factor and a slow (e.g. attention) factor you should see variance explained in the 20-60% range. In my paradigm I only get about 6-12% likely because I have two fast alternating events (stimulus intensity and probability). 

    On the model fit question, I think this comes down to what the variational free energy parameter is actually optimizing. I am not an expert on the calculus behind VFE, but i've been to the DCM workshops and tried to understand the theory as much as my limited calc background allows. Essential VFE is a measure that weights the fit of the model (how well the model predicts the data, in a bayesian posterior sense) by the model complexity. I found this bit of Karl's response helpful:

    "Generally speaking, a model with a poor fit can have more
    evidence than a model with a good fit. For example, if you give a
    model pure measurement noise, a good model should properly identify
    that there is noise and no signal. This model will be better than a
    model that tries to (over) fit noisy fluctuations."

    So the model evidence accounts for fit, but goes beyond it. As Karl points out, a model with perfect fit can be a very poor model, so the VFE is an attempt to balance these things. Beyond that all I can say is that I was taught to always look at the variance explained as a diagnostic step - generally the rule of thumb here is that if your variance explained is really low, you probably made a mistake somewhere or the paradigm is very poorly optimized for DCM. I think in a practical sense, Lohmann have a point here, as there are diagnostic tools (like spm_dcm_explore) but no real guidelines for using them (when to decide you've mucked up the DCM). I think strictly speaking the VFE does the job, but only in the (perhaps too ideal) world where the models make sense.

    I found this blog really useful for understanding why DCM sometimes fails to give a good variance explained for event related paradigms:
    http://spm.martinpyka.de/?p=81
    spm.martinpyka.de
    What follows is a rough but hopefully didactically useful introduction into the ...See More
  • Russ Poldrack this was with fMRI data (with Marta Isabel Garrido) - but it was with fast ER designs, which are pretty much all we do in my lab, so that might explain part of the poor fit. Karl's point about penalizing model complexity is totally reasonable, but I'm not sure that it really has anything to do with VFE per se -VFE is just one of many ways to penalize for model complexity. (of course, it's Bayesian, so it must be better 
  • Daniel Handwerker The other trouble with the current DCM approaches is that they rely heavily on assumptions of what hemodynamic responses should be - including using the balloon model in a way it was never intended to be used. As part of a commentary on hemodynamic variation (Neuroimage 2012), I ran a small side analysis that showed how a modest, but very believable, difference in the size of the post-peak undershoot can flip the estimated direction of causality using DCM. This was a bit of a toy analysis, but it highlights a real concern how assumptions in building low-level parts of SPM's DCM model can really affect results.
  • Micah Allen Daniel, i'm not sure i'd agree that is a limitation. The nice thing about DCM is since it is built around a Bayesian hypothesis testing architecture, any competing HRF can be substituted for another and the resulting model estimates compared. So you could easily run a DCM with your HRF vs the canonical - if yours wins it would be a good argument for updating the stock model. The HRF part of DCM is totally modular, so a power user should find it easy to substitute a competing model (or multiple competing models). This point was made repeatedly at the DCM workshop in Paris last year.
  • Tal Yarkoni Micah, I think you're grossly underestimating the complexities involved. Take the problem of selecting ROIs: you say it's incumbent on the researcher to correctly interpret the mass univariate results. But what does that mean in practice? Estimation uncertainty at any given voxel is high; if you choose an ROI based on passing some p value threshold, you will often have regions in which the confidence interval's lower bound is very close to zero. With small samples, you will routinely miss regions that are actually quite important at the population level, and routinely include other regions that are non-zero but probably should not be included. If you use very large samples, on the other hand, then everything is statistically significant, so now you have the problem of deciding which parts of the brain should be part of your causal model and which shouldn't based on some other set of criteria. If you take what you just said to its conclusion, people probably shouldn't fit DCM models to small datasets period, since the univariate associations are known to be shaky.

    Even if you settle on a set of ROIs, the problem of definition is not just one of avoiding "haphazard" selection; e.g., Steve Smith's work in NeuroImage nicely showed that even relatively little mixing between different nodes' timeseries will seriously muck up estimation--and that's with better behaved network modeling techniques, in a case where we know what the discrete nodes are a priori (because it's a simulation), and using network structures that are much more orderly than real data are likely to be (Smith et al's networks are very sparse). In the real world you typically don't know any of this; for example, in task-related activation maps, the entire dorsal medial surface of the brain is often active to some degree, and there is no good reason to, say, split it into 4 different nodes based on one threshold versus 2 nodes at another--even though this is the kind of choice that can produce wildly different results in a structural model.

    As for the hemodynamic issue: the problem is not so much that the canonical HRF is wrong (though of course we know it often is--and systematically so, in that it differs reliably across brain regions)--it's that you compound all of its problems when your modeling depends on deconvolved estimates. It's one thing to say that there is X amount of activation at a given brain region when you know that your model is likely to be wrong to some degree, and quite another to say that you can estimate complex causal interactions between 12 different nodes when the stability of that model depends on the same HRF fitting the data well in all cases.

    As to the general idea that DCM depends on strong hypotheses; this sounds great in principle, but the problem is that there are so many degrees of freedom available to the user that it is rarely clear what constitutes a disconfirmation of the hypothesis versus a "mistake" in one's initial assumptions. Of course to some degree this is a problem when doing research of any kind, but it's grossly compounded when the space of models is effectively infinite and the relationship between the model space and the hypothesis space is quite loose (in the sense that there are many, many, many network models that would appear consistent with just about any theoretical story one can tell).

    Mind you, this is an empirical question, at least in the sense that one could presumably quantify the effect of various "trivial" choices on DCM results. Take the model you mention with 6-7 nodes: I would love to know what happens if you systematically: (a) fit models that subsample from those nodes; (b) add nodes from other ROIs that didn't meet the same level of stringency; (c) define ROIs based on different statistical criteria (remember that surviving correction is a totally arbitrary criterion as it's sample size dependent); (d) randomly vary the edges (for all of the above). The prediction here is that there should be a marked difference in terms of model fit between the models you favor and the models generated by randomly permuting some of these factors--or, that when other models fit better, they should be theoretically consistent in terms of interpretation with the favored model. Is this generally the case? Has anyone demonstrated this?
  • Rajeev Raizada I read the above discussion just now, and found it very interesting indeed. I have never tried playing with DCM, but I have worked in neural modeling in the computational neuroscience / neural circuits sense. In general, I am skeptical of complex and sophisticated models, especially models which congratulate themselves on their complexity and sophistication, or, in the case of some of the models mentioned immediately above, models which congratulate themselves on the sophistication of their self-quantification of complexity. 

    A question for the DCM/GCM-aficionados out there: is there a single instance of such approaches generating a novel insight into brain function which was later independently validated by a different method? It looks to me as though there are a lot of instances of such approaches producing interesting-looking outputs which seem reasonable given what we think we know about the brain. But the road to hell in science is paved with interesting and reasonable-sounding stories.

    This last line is partly troll-bait, but I'll throw it out there anyway as I think it might be a valid comparison: are DCM/GCM approaches a bit like the evolutionary psychology of fMRI? Sources of interesting just-so stories?
  • Micah Allen Out of the office at dinner now so I'm afraid I must leave this debate for now. You make dinner excellent points Tal though I'm not sure I agree with the picture you paint of fMRI effects bring quite so arbitrary. At my former center we stuck to an N = 30 guideline and found this to be an excellent compromise, neither under nor over powered. In my oddball paradigm I get extremely anatomically constrained activations that fit well with the literature. I extract 6mm spherical VOIs from each peak. This seems like a pretty reasonable way to characterize the overall effect, but I do think the analysis you suggest would interesting in any context DCM or otherwise. Sorry so brief- on my phone at dinner!
  • Micah Allen Rajeev Raizada a good question but there is a lot of ongoing intracortical and multimodal validation work being done with DCM. Check out Rosalyn Moran's recent papers 
  • Rajeev Raizada Sounds interesting. Are you talking about this paper, which finds that boosting acetylcholine increases the size of the mismatch negativity?http://www.ncbi.nlm.nih.gov/pubmed/23658161 An interesting result, but it's not entirely clear to me that the theoretical overlay of predictive-coding and DCM adds a great deal, or that the empirical result adds much support to the theory. After all, this paper from 2001 already showed the converse result (reduced ACh gives reduced MMN): http://www.ncbi.nlm.nih.gov/pubmed/11467911 , and there is a ton of evidence showing that ACh increases cortical signal-to-noise. However, I may be focussing on a different paper than you were referring to.
    www.ncbi.nlm.nih.gov
    PubMed comprises more than 23 million citations for biomedical literature from M...See More
  • Daniel Handwerker As Tal notes, the issue with hemodynamic variation is that it varies quite a bit around the brain and there is no existing model in a Baysean or any other framework that can solve this degree of variation. This isn't a problem if one's analysis is robust to much of the variation, but the deconvolution step in DCM amplifies its sensitivity to the selected hemodynamic model. If two brain regions having slightly different relative undershoots is enough to make the model fail, then when can we trust that it works?

    Another way of saying that a power user can make their own hemodynamic model is to say that the model that is default with the SPM DCM software doesn't work in real-world situations, but that doesn't preclude someone in the future from making a model that does work. This might be true, but it does little to increase confidence in the accuracy of current DCM studies. 

    As others have noted, simpler causality measures often have their assumptions more out in the open and someone can design a study that is robust to those assumptions. If a study was designed well enough that it could work within the complex assumptions of DCM, I can't think of a situation where it wouldn't also meet the assumptions of simpler approaches.
  • Micah Allen No, the option is there for those who want to show that there is a better model. The canonical balloon HRF enjoys a great deal of robustness, which is why its the function of choice in nearly all neuroimaging packages. Further DCM has been shown to be very robust to lag differences of up to 2 seconds. I hope you guys at not out there rejecting DCM papers on these basic principles without at least reading the actual validation papers behind it. Your toy example shows that DCM is sensitive to the HRF of choice, not that the canonical is an inappropriate model. If there is that big of a smoking gun problem with DCM, I'd probably suggest you publish to that effect to prevent a lot of people from wasting time!
  • Micah Allen Just to try to close on a positive note, I find this review by Stephen Smith to be exceptionally balanced, cutting through much of the hyperbole surrounding different connectivity methods. I think he really does a great job pointing out the natural strengths and weaknesses of each of the available approaches, advocating for a balanced viewpoint. Also don't miss the excellent foot note where he describes views from 'scientist A and scientist B' - pretty obvious who he's referring to there  
    http://www.sciencedirect.com/.../pii/S1053811912000390#
  • Marta Isabel Garrido This is one of the best validation papers showing robustness of DCM for fMRI (and also how GC fails terribly) http://www.ncbi.nlm.nih.gov/pubmed/19108604
    www.ncbi.nlm.nih.gov
    PubMed comprises more than 23 million citations for biomedical literature from M...See More
  • Daniel Handwerker Micah, when I say that a fundamental assumption of the current implementation of DCM fails under very realistic conditions to such an extent that it will flip the result, improving DCM is not my only option. The other option is to not personally use DCM and treat DCM findings which rest on these fundamental assumptions with a good bit of skepticism. If I had a specific project that would clearly benefit from the DCM approach, I might reassess my stance, but until then, it's the job of DCM users to make the method/interpretations more robust.
    I'll also note that the robustness of the canonical HRF is analysis dependent. My 2004 Neuroimage paper showed that it's robust for finding significance maps using a GLM, but it is less robust if you're taking magnitude estimates into a group analysis. Still, the robustness of the canonical HRF in linear regression based studies has been shown to be pretty good. That doesn't mean the same model is robust when used as part of a completely different analysis method. Any causality measure is probably going to be more sensitive to GLM shape compared to intravoxel statistical approaches. My review that I mentioned earlier ( http://www.sciencedirect.com/.../pii/S1053811912001929 ) is a published example showing how DCM can fail. It's an example rather than a full examination of the method's limitations, but it's enough to cause me some concern.
  • Nikolaus Kriegeskorte this is a fascinating thread. i'm left with three thoughts.
    (1) the paper by webb et al. does not seem to invalidate granger causality inferences based on comparisons between different experimental conditions. it would be good to hear Alard Roebroeck's
    take on these issues. (2) it would be good to have an fMRI causality modelling competition in which several simulated data sets (with known but unrevealed ground truth and realistic complexities) are analysed by multiple groups with multiple techniques. (3) the only thing that puts the proof in the pudding for me is prediction. in decoding analyses, for example, the use of independent test sets (crossvalidation) ensures that incorrect assumptions make us more likely to accept the null hypothesis. what is the equivalent of this in the realm of causal analyses?
    16 hours ago · Edited · Unlike · 2
  • Russ Poldrack Niko - agreed, the only reasonable use of Granger causality analysis with fMRI that I have seen is the work by Roebroek that showed differences between conditions in G-causality within the same region, which mostly obviates the issues regarding HRFs (unless you think the latency is changing with amplitude). if only I was FB friends with Alard! And I second both your call for a competition (though the devil is in the details of the simulated data) and the ultimate utility of crossvalidation.
  • Jack Van Horn Niko: something like your item #2 as a challenge for the OHBM Hackathon.#justsayin
    15 hours ago · Like · 2
  • Nikolaus Kriegeskorte jack, that would be great. for the next meeting, we have an organising team with a plan already. could consider this for hawaii. it would be good to hear opinions from Klaas and Karl (whom i'm missing on facebook), Alard Roebroeck, and Stephen Smith -- as well as the commentators above.
    15 hours ago · Like · 1
  • Nikolaus Kriegeskorte Russ Poldrack is right that the devil is in the details of the simulated data. each method may prevail when its model matches the one that generated the data. it will therefore be key to include realistic complications of the type described by tal, which none of the methods address. the goal could be to decide about each of a set of causal claims that either do or do not hold in the data-generating ground-truth model.
  • Nikolaus Kriegeskorte I'd like to hear Rik Henson's take on this entire thread.
  • Micah Allen This has been a very informative thread for me - wish Facebook posts could be storified as this is an important debate. I would love to see it formally continued at OHBM. I will try to get Karl's opinion on these issues at the next methods meeting. Over beers last night several members of that group expressed that some of this validation was underway, and that DCM is undergoing significant revision to address many of these issues. Still it's clear that there has not been enough discussion. Also as a side note I agree that the GCM paper in my blog post does not necessarily implicate between condition differences. I think the answer to some of these issues is inevitably going to come down to experimental design, as probably both GCM and DCM can be more or less robust to vascular confounds depending on the timing and nature of your paradigm. It would be nice to know exactly what those constraints are.
    5 hours ago via mobile · Edited · Like
  • Jack Van Horn This brings to mind early 1990's concerns about the effects that vasculature and draining veins, in particular, had on measured regional BOLD activation. This was considered a real issue by some and a potential deal breaker for the future of fMRI over PET. But others felt that the HRF actually saved the day since it would be modulated by neural activity under cognitive task conditions. The issue kind of got swept under the rug for the last 15 years or so. Interesting that this is now re-emerging in the realm of functional connectivity, resting-state, and notably Granger causality. Could it be that auto-correlative Granger modeling is doing exactly what it is supposed to do? And perhaps it has simply been our poor understanding of its implications relative to actual hemodynamics that is finally catching up to us? I look forward to seeing the further discussions.

Saturday, May 4, 2013

Crowdsourcing my next fMRI task

For those of you who have been following my self-tracking study, I am happy to announce that we are once again collecting data as of April 30.  I had a followup audiometry exam this week which showed that the poor reading in early March appears to have been a fluke, and that my hearing does not seem to have changed since the beginning of the study.

We have decided to make a number of changes to the MRI protocol based on our analyses of the first 6 months of data. In particular, we are going to reduce the frequency of the structural and DTI acquisitions (because, as expected, we don't see much in the way of changes over time).  In addition, after acquisition of 14 sessions on a working memory task, I have decided to reduce its frequency and add another task fMRI paradigm that will be collected once a week.

My question for you is: What should I do next?  I have several things in mind, but I want to see if the community can do even better.  If you have an idea for task fMRI paradigm where a large number of sessions from a single individual would be useful, please email me (poldrack at utexas dot edu) with a brief proposal that outlines specifically why it would be interesting to perform the task repeatedly on a single person and how many sessions it would require.  I will work with the winner to implement the task, and that person will of course be offered coauthorship on any papers that include those data.  There are two requirements:

  • Each session of the task must be accomplished within a single 10-minute imaging run.
  • The task cannot require any special response or stimulation devices (for stimulation we have video projection and headphones; for response, we have several button boxes available as well as eye tracking, noise-cancelling microphone, and physiological monitoring).
I look forward to hearing your ideas! 



Friday, May 3, 2013

The dimensional approach to studying mental illness

The NIHM Director Tom Insel turned a lot of heads recently with his announcement that NIMH-funded research will no longer focus on DSM-based diagnostic categories, and instead will focus on understanding the dimensions of mental function that underlie mental health disorders and often cut across multiple diagnoses.  This new focus will be centered around the Research Domain Criteria (RDoC), which comprises a set of psychological constructs that are thought to represent the most important dimensions that are relevant to mental health disorders. This is great news, as it has become clear (particularly from recent genetic research) that diagnostic categories, while being statistically reliable, do not have a great deal of biological reality. Dr. Insel lays out the problem nicely:

Unlike our definitions of ischemic heart disease, lymphoma, or AIDS, the DSM diagnoses are based on a consensus about clusters of clinical symptoms, not any objective laboratory measure. In the rest of medicine, this would be equivalent to creating diagnostic systems based on the nature of chest pain or the quality of fever. Indeed, symptom-based diagnosis, once common in other areas of medicine, has been largely replaced in the past half century as we have understood that symptoms alone rarely indicate the best choice of treatment.

We have long been interested in this dimensional approach; in particular, it was the basis for the Consortium for Neuropsychiatic Phenomics developed by my colleague Robert Bilder at UCLA several years ago, in which we are examining two specific domains of mental function (memory and executive function) across both healthy individuals and people with several different psychiatric diagnoses (schizophrenia, bipolar disorder, and ADHD).  This work is still ongoing but should start to bear fruit in the next year as the neuroimaging and genetic data analyses are completed.  It's great news that more work like this will be funded in the future.

Mining for dimensions

Genetic analyses have been able to take advantage of large numbers of subjects to identify the genetic overlap between different psychiatric diagnostic groups.  Unfortunately we don't have those kinds of datasets (yet) for neuroimaging studies.  In a paper published in PLOS Computational Biology last year, we asked whether we could use the Neurosynth meta-analytic database as a proxy for such data, in order to see if we can identify how different psychiatric disorders are clustered together in terms of brain function.  We first used topic modeling to identify groupings of terms related to psychiatric and neurological disorders in the full text of a set of about 4,400 published papers.  In some cases these were specific to single disorders, but in many cases they reflected overlapping disorders (such as alcoholism and antisocial personality disorder, or schizophrenia and bipolar disorder).  We then created "topic maps" for each of these by looking at which regions in the brain showed activity that was correlated with the presence of each topic in each paper (using the activation coordinates automatically extracted by Neurosynth).  Here are some examples of these topic maps:



We then were able to ask the fundamental question that is raised by the dimensional approach: How are different diagnoses related in terms of their neural underpinnings.  It's important to keep in mind that we are not directly examining data from different diagnostic groups; instead, we are examining the neuroimaging data that are associated with the presence of those diagnostic labels across papers.  Nonetheless, the data have the potential to give us insight into how different disorders relate with regard to neural activity patterns observed in the literature.  Here is what the clustering looked like across disorders:


(the abbreviations are: APH: aphasia, DLX:dyslexia, SLI: specific language impairment, DA: drug abuse, AD:Alzheimer's disease, DEP:depressive disorder, MDD:major depressive disorder, ANX:anxiety disorder, PAN: panic disorder, BPD: bipolar disorder, CD: conduct disorder, GAM: gambling, MD: mood disorder, PD: Parkinson's disease, OCD: obsessive compulsive disorder, PHO: phobia, EAT: eating disorder, SZ: schizophrenia, OBE: obesity, COC: cocaine related disorder, PSY: psychotic disorder, PAR: paranoid disorder, SZTY: schizotypal personality disorder, TIC: tic disorder, ALC: alcoholism, ALX: alexia, ADD: attention deficit disorder, AMN: amnesia, AUT: autism, ASP: Asperger syndrome.)

This clustering is very interesting in that it shows that there are four major branches of mental health disorders that can be identified on the basis of brain activity coordinates from published papers: language disorders (green), mood/anxiety disorders and drug abuse (orange), psychotic and externalizing disorders (yellow), and autism and memory disorders (purple).  The fact that this worked with meta-analytic data gives us great hope that once it is applied to subject-level data, it will provide a great deal of power to identify the dimensions along which psychiatric disorders vary.

How reliable are the dimensional descriptions?

The move towards RDoC as a basis for mental health research also raises the question of reliability.  The strength of the DSM was that its diagnoses were reliable, in the sense that two different psychiatrists, shown the same symptoms, are highly likely to make the same diagnosis.  How will we ensure that the mappings from behavior to psychological dimensions are equally reliable?  This is a problem that we have long struggled with, and in particular was a driver for the development of our Cognitive Atlas project.  This project aims to provide a more formal basis for the many psychological constructs that are used by researchers but rarely defined in an explicit way.  It makes a clear distinction between mental concepts (i.e. the unobservable psychological constructs that researchers want to measure) and tasks (i.e., the behavioral tests used to measure to those constructs), and most importantly, provides a way to specify the relations between those two levels.  For example, here is the page representing the concept of "response inhibition":



This page provides a specific definition of the concept along with links to many different tasks that are thought to measure the concept, such as the stop signal task:


Recently we added a new function to the Cognitive Atlas, called Collections.  A collection is a set of concepts and relations that together form a larger theoretic framework; it can also be used to identify sets of tasks that are included together in task batteries.  We have started to implement the various RDoC frameworks as Collections; for example, the Working Memory Matrix:



There is much more work to do, and we hope that researchers interested in RDoC will help by adding their favorite tasks and concepts to the appropriate matrices.  In the long run we are hopeful that formal frameworks like the Cognitive Atlas will come to play the same role for dimensionally-driven research that the DSM has played for diagnosis-driven research, providing a basis for reliable specification of the relations between psychological dimensions and behavioral tasks.



Friday, April 19, 2013

On "healthy choices"


This week, I received an email describing new nutritional recommendations at one of the campus eateries:
Please come by O's Campus Cafe over the next few weeks to see how they are making healthy eating easier!
O's Campus Cafe has analyzed their menu and labeled their foods in order to make it easier for you to make the healthiest choices.  All foods have been given a red, yellow, or green rating based on how nutrient dense they are. Foods that have received a "Red" rating have very little positive nutrient content, are higher in unhealthy nutrients, and should be consumed minimally.  Foods that have received a "Yellow" rating have some positive nutrient content, some unhealthy nutrients, and should be consumed less often.  Foods that have received a "Green" are high in positive nutrient content, have little to no unhealthy nutrients, and can be consumed frequently.  The healthy nutrients evaluated in these developing these ratings are protein, fiber, unsaturated fat, vitamins A, C, E, B-12, thiamin, riboflavin, and folate, and minerals calcium, magnesium, iron, and potassium.  The nutrients considered unhealthy are saturated fat, cholesterol, and sodium. 
I would hope that guidelines regarding "healthy" choices at a major research university would be backed up with evidence, but it turns out that they were simply based on an off-the-shelf rating system (the Nutrient Rich Food Index). To see whether they are supported by actual evidence, I used PubMed to look for relevant results from randomized controlled trials; I avoided epidemiological studies given all of the difficulties in making solid conclusions from nutritional epidemiology research (e.g. http://www.nytimes.com/2007/09/16/magazine/16epidemiology-t.html). Here is what I found regarding the supposedly "unhealthy" nutrients:
  • Reduction of saturated fat intake was associated with reduced cardiovascular events but not with any changes in all-cause or cardiovascular mortality (Hooper et al., 2012). A nice review of lipid metabolism by Lands (2008) highlights how the search for mechanisms by which saturated fats might be harmful has failed: "Fifty years later [after starting his research in the 1960's], I still cannot cite a definite mechanism or mediator by which saturated fat is shown to kill people."

  • A review of the effects of dietary sodium reduction in randomized controlled trials showed "no strong evidence of any effect of salt reduction" on all-cause mortality (Taylor et al., 2011).
  • I searched deeply on PubMed but was unable to find any systematic reviews of the effects of dietary cholesterol reduction on mortality (please let me know if you know of any); there are lots of studies looking at cholesterol reduction using statins, but that doesn't really inform the question of dietary cholesterol reduction.  However, given that serum cholesterol levels bear very little relation to dietary cholesterol intake (Gertler et al., 1950), the labeling of dietary cholesterol as "unhealthy" seems unsupported by evidence. 

The first two reviews listed above come from the Cochrane Collaboration, which is generally viewed as a highly objective source for systematic reviews in evidence-based medicine.

What about the supposed "healthy" choices?

I also looked for data relevant to the the health benefits of the supposedly healthy choices:
  • A recent report from a randomized controlled trial showed that replacement of saturated fats with omega-6 linoleic acid (an unsaturated, and thus "healthy" fat) was associated with increased, rather than decreased all-cause mortality (Ramsden et al., 2013). This makes sense given the association of omega-6 fatty acids with inflammation (e.g. Lands, 2012)  and the powerful role that inflammation plays in many diseases.

  • Increased fiber intake in the DART (diet and reinfarction trial) of individuals who had already suffered a heart attack was associated with no effects on all-cause mortality after 10 years.(Ness et al., 2002).  An earlier Cochrane review also showed no effects of fiber intake on incidence or recurrence of colon cancer (Asano & McLeod, 2002).

I don't claim to be an expert in this domain, but the best evidence available seems to point towards the lack of any scientific validity of these "health" labels.  

Isn't it better than nothing?

One might ask whether it's still better to have some guidance that pushes people to try to eat more healthily, even if it's not supported by science.  If the recommendations were made with full disclosure of the lack of scientific evidence (ala the disclaimer on dietary supplements) then I would be ok with that. Instead, the recommendations are made in the context of a "nutritionism" that pretends that we will be healthy if we just get the right mix of healthy nutrients and avoid the bad ones, and also pretends to know which are which.  I personally would rather have no advice than unsupported pseudoadvice.

At the same time, it's clear from the obesity epidemic that people need help in making better food choices, so what should we do instead?  I would propose implementing a simple rating system based the first of Michael Pollan's food rules: "Eat food".  Anything that has been freshly picked, harvested, or slaughtered would get a "healthy rating" and anything that comes from a box, can, or bag would get an "unhealthy" rating. It would be much easier than computing micronutrient values, and I challenge anyone to show that it would be any less objectively healthy than the proposed "healthy choices" which would label a fresh grass-fed steak as less healthy than a pile of french fries fried in corn oil.

Saturday, April 6, 2013

How well can we predict future criminal acts from fMRI data?


A paper recently published in PNAS by Aharoni et al. entitled "Neuroprediction of future arrest" has claimed to demonstrate that future criminal acts can be predicted using fMRI data.  In the study, the group performed fMRI on 96 individuals who had previously been incarcerated, using a go/no-go task.  They then followed up the individuals (up to four years after release) and recorded whether they had been rearrested.  A survival model was used to model the likelihood of being re-arrested, which showed that activation in the dorsal anterior cingulate cortex (dACC) during the go/no-go task was associated with rearrest, such that individuals with higher levels of dACC activity during the task were less likely to be rearrested.  This fits with the idea that the dACC is involved in cognitive control, and that cognitive control is important for controlling impulses that might land one back in jail.  For example, using a median split of dACC activity, they found that the upper half had a rearrest rate of 46% while the lower half had a rearrest rate of 60%.  Survival models also showed that dACC was the only variable amongst a number tested that had a significant relation to rearrest.

This is a very impressive study, made even more so by the fact that the authors released the data for the tested variables (in spreadsheet form) with the paper.  However, there is one critical shortcoming to the analyses reported in the paper, which is that they do not examine out-of-sample predictive accuracy.  As I have pointed out recently, statistical relationships within a sample generally provide an overly optimistic estimate of the ability to generalize to new samples.  In order to be able to claim that one can "predict" in a real-world sense, one has to validate the predictive accuracy of the technique on out-of-sample data.

With the help of Jeanette Mumford (my local statistical guru), I took the data from the Aharoni paper and examined the ability to predict rearrest on out-of-sample data using crossvalidation; the code and data for this analysis are available at https://github.com/poldrack/criminalprediction.  The proper way to model the data is using a survival model that can deal with censored observations (since subjects differed in how long they were followed).  We did this in R using the Cox regression model from the R rms library.  We replicated the reported finding of a significant effect of dACC activation on rearrest in the Cox model, with parameter estimates matching those reported in the paper, suggesting to me that we had correctly replicated their analysis.  

We examined predictive accuracy using the pec library for R, which generates out-of-sample prediction error curves for survival models.  We used 10-fold crossvalidation to estimate the prediction error, and ran this 100 times to assess the variability of the prediction error estimates. The figure below shows the prediction error as a function of time for the reference model (which simply estimates a single survival curve for the whole group) in black, and the model including dACC activation as a predictor in green; the thick lines represent the mean prediction error across the 100 crossvalidation runs, and the light lines represent the curve for each individual run.  



This analysis shows that there is a slight benefit to out-of-sample prediction of future rearrest using dACC activation, particularly in the period from 20 to 48 months after release.  However, this added prediction ability is exceedingly small; if we take the integrated Brier score across the period of 0-48 months, which is a metric for assessment of probabilistic predictions (taking the value of 0 for perfect predictions and 1 for completely inaccurate predictions), we see that the score for the reference model is 0.214 and the score for the model with dACC as a predictor is 0.207. We found slightly improved prediction (integrated Brier score of 0.203) if we also added Age alongside dACC as a predictor.  

The take-away message from this analysis is that fMRI can indeed provide information relevant to whether an individual will be rearrested for a crime.  However, this added predictability is exceedingly small, and we don't know whether there are other (unmeasured) demographic or behavioral measures that might provide similar predictive power.  In addition, these analyses highlight the importance of using out-of-sample prediction analyses whenever one makes a claim about the predictive ability of neuroimaging data for any outcome.  We are currently preparing a manuscript that will address the issue of "neuroprediction" in greater detail.  

Wednesday, March 13, 2013

My adventures in self-quantification


In September 2012 I began a project to characterize how my own brain function and metabolism fluctuate over the course of an entire year.  This has involved MRI scans three times a week along with blood draws once a week and daily tracking of a large set of potentially interesting variables.  This post is the first installment in my story about the experience.

First, the motivation.  In the last couple of years I have become very interested in understanding the dynamics of brain function over a days-months timescale and how they relate to cognitive function and bodily metabolism.  This interest has been spurred by a number of influences, such as my growing interest in nutrition and its relation to brain function as well as my ongoing interest in better understanding psychiatric disorders.  Once I started thinking about the issue, it became very clear that there were basically no data in existence that provide any insight into how the function of an individual's brain fluctuates over such a relatively long time course. This is probably not surprising, because doing studies with volunteers that require repeated testing over a long period of time is very challenging.  

At some point in 2011 it dawned upon me that I should try to bootstrap such a study by collecting data from myself.  There were several inspirations for this idea.  First was Michael Snyder's "integrated personal omics" study, published in Cell in 2011, in which he repeatedly collected blood from himself and performed a broad set of "omics" measures on his samples, which provided some interesting insights into the temporal dynamics of metabolic function.  Second was my interaction with Laurie Frick, who is the artist-in-residence at the UT Imaging Research Center.  Laurie's work is based on patterns that she finds in data obtained by self-tracking, and she is deeply enmeshed in the Quantified Self movement (see her excellent TEDx talk).  Talking to her got me increasingly interested in tracking a broader set of data about myself. The very fun book Smoking Ears and Screaming Teeth also convinced me that self-experimentation is not (completely) crazy, and actually has long been an important tool for scientific discovery.

In early 2012 I began hatching a plan to collect a broad set of data on myself.  It was essential that the all aspects of data collection were as consistent as possible in order to minimize extraneous variability in the data (such as time of day effects).  I ended up settling on a schedule of three MRI scanning sessions a week, at consistent times of day and days of the week (one afternoon and two mornings every week).  Each of the MRI scanning sessions includes a resting state fMRI scan, which will allow us to assess how functional connectivity between brain regions fluctuates over time.  In addition, once a week I perform other scans, including structural MRI (T1- and T2-weighted), diffusion tensor MRI (to assess white matter connectivity), and task fMRI (using a working memory task with faces, scenes, and chinese characters).  

I also wanted to collect biological samples in order to measure the relation between bodily metabolism and brain function.  Working with some molecular biologists here at UT (along with helpful input from the Snyder lab at Stanford), we developed a protocol in which I have 20 ml of blood drawn once a week (while fasting, immediately after one of the morning MRI scans). This sample is then processed to extract RNA, white blood cells, and plasma, all of which are frozen for later analysis.  This will let us examine many different aspects of metabolism, including gene expression (via RNA sequencing), metabolomic and proteomic analyses, and other potential analyses to relate metabolism to brain function.

Finally, I realized that the dataset would be most useful if I also collected as much data as possible about my daily life activities.  Working with Zack Simpson, we developed a self-tracking app using the Appsoma framework, which allows me to easily complete surveys every morning and evening and after every MRI scan.  These data are automatically fed into a web database which is the central repository for all of the self-tracking data in the study other than MRI and biological analyses.  Some of the things that I track daily include:
- blood pressure and weight (using a FitBit Aria wireless scale)
- foods eaten, alcohol intake, and supplements/medicines taken
- exercise, time spent outdoors, and physical soreness
- a free-text log of daily events
- sleep quality (assessed both by subjective report and using a ZEO sleep monitor)
After every scan I also complete a mood questionnaire and also provide a structured report of what I was thinking about during the resting state fMRI scan.  I should note that other than the addition of all of these tracking activities, I have done my best to keep my life as consistent as possible and have avoided any other major lifestyle changes.

With this plan in place, we began data collection on September 25, 2012.  We treated the first month as a pilot period, and made some changes to the imaging protocol to optimize data collection, beginning the production period on October 22, 2012.  In total so far we have collected 20 blood samples and 55 MRI scanning sessions.  Members of the research team have started analyzing the data, though I have made every effort not to expose myself to the results of any analyses that examine changes over time, because I don't want the results to feed back and change my behavior.  

When I describe this study, many people ask if I am worried about being exposed to MRI scanning so often.  My answer has been "no", at least not with regard to the magnetic fields involved in MRI; there is no evidence of lasting effects of MRI exposure (though of course we can't ever prove that something is safe).  However, soon after the study began it became clear that there was a side effect that I had to worry about, which is the intense noise of the MRI scanner.  I have long suffered from tinnitus (which I attribute to too many loud rock shows as a youngster without ear plugs), and within the two weeks of scanning I noticed that my tinnitus was increasing.  For this reason, I went to the UT Speech and Hearing Center and had my hearing tested.  I had never had my hearing tested as an adult, but I was not terribly shocked to find out that I had quite significant high frequency hearing loss.  Because I don't want to damage my hearing any further, I have continued to get tested each month.  The results had been fairly stable until early March, when they showed about a slight worsening at 6000 Hz (consistent with a subjective increase in tinnitus around the same time).  For this reason, I am taking the month of March off of scanning, and will have my hearing re-tested at the beginning of April before resuming the scans.

We will also make some changes to the MRI protocol to reduce scanner noise.  We have an OptoAcoustics noise canceling headphone system in place at our imaging center that works quite well to reduce the noise of the functional MRI scans, so those can continue without much danger.  However, we will likely discontinue some of the other scans (such as gradient field maps, which are useful but not necessary) and greatly reduce the frequency of others that we don't expect to change much over time (including the anatomical and diffusion scans), because those scans are not compatible with the noise cancellation system.  I am hopeful that with these changes I can continue scanning without danger of further hearing damage, while still collecting a very useful dataset.  

Assuming that I am able to continue scanning, I plan to collect 50 weeks worth of usable data, which should provide sufficient power for an initial set of analyses.  This will likely take though the end of 2013 due to travel and other events that will interfere with data collection during some weeks.  Once we have completed our initial set of analyses, nearly all of the data will be made available to other researchers, which I hope will help spur new analyses.  I'll keep you all posted as the study moves along.