Returning to Robustness

In Discussion on March 7, 2010 by Jenny

The article from Orzack and Sober that we read this week returned to a lot of questions that were never answered in our first discussion of robustness.  That is, why should we think that a robustness analysis works?

Orzack and Sober deliver (at least) two hearty blows to Levins’s account of robustness:

1. “If we know that each of the models is false (each is a ‘lie’), then it is unclear why the fact that R [i.e. the robust theorem] is implied by all of them is evidence that R is true” (538).

2. For a robustness analysis to work, the models need to be independent.  Otherwise the ‘robustness’ we find may simply reflect a commonality of the models’ frameworks, rather than a truth regarding something the frameworks describe.  But the models cannot be logically independent nor does it make sense to talk about their statistical independence (539-40).  So robustness analyses do not work.

Then the question is, why should we still think robustness analysis has any confirmation or explanatory power, given these criticisms?



9 Responses to “Returning to Robustness”

  1. Jenny,

    A thought regarding O&S’s first point. Certainly, one could take robustness to say that multiple models predicting the same property R simply is evidence that the robust theorem R is true. But can it be tweaked a bit to say something weaker? What if the implication is “If models [p], [q], [r], etc. all imply R, then R is either true or a collective artifact of the models”?

    While it’s logically possible that each model is “lying” in the same way, the odds of this being the case would seem to lessen as each new model is added. We could never say with certainty that R is true, but we could say it with increasing confidence. Is there a point in modeling and robustness where we are confident enough to accept a theorem, leaving open the possibility of a defeater?

  2. Jake,

    Thanks for your response. A couple quick points on it. You write that we could tweak O&S’s 1st point to say that “If models [p], [q], [r], etc. all imply R, then R is either true or a collective artifact of the models.”

    To me it just sounds like you’re saying that if [p], [q], and [r] imply R, then R is either true or false–a fact that is trivially true. To eliminate this problem, it needs to be explained why adding to our account that R might be a collective artifact of the models does more than just says that R could be false. Any ideas on this?

    You also write that the odds that the models are “lying” decreases when each new model is added. Assuming that the models are independent (which O&S argue is probably not the case), then it is true that adding new models to the mix could increase the chances of R being true. But the question is, “by how much?”

    Levins writes (somewhere) that it is impossible to figure out all of the possible models for a phenomenon, implying that there are likely thousands of possible ways of modeling a given occurrence. If this is so, then adding more models into the mix might merely increase our probability of being right by, say, a thousandth of a percent. That doesn’t seem particularly helpful. But perhaps, if you add enough models, this would increase our probability of being right.

    The question is whether the number of models that would have to be added to have such confirmatory power is high enough to be so cumbersome that scientists find using a robustness analysis practically useless.


  3. I shall start with a bold statement: science knows nothing and all it produces is most likely false. Predictive validity has nothing to do with ultimate metaphysical accuracy. The world is likely much stranger and more complex than we are currently entertaining.

    That said, I think Weisberg covers robust analysis well. “The procedure itself does not confirm hypotheses[.]” p.643. “Levins was not offering an alternative to empirical confirmation; rather he was explaining a procedure used in conjunction with empirical confirmation in situations where one is relying on highly idealized models.” p.642.

    The models themselves have different assumptions; that is different parts are idealized, and in this sense they are independent of one another. The common nonidealized part was generated from empirical observation of the phenomena in question. Each idealized model is necessarily ‘false’ because it has simplified or assumed away complex parts of the phenomena to make the model manageable.

    If we then find some property, outcome, or dynamic that is common across these different idealized models, there is likely some common causal structure (namely the common nonidealized part) that gives rise to the property in question.

    This is, of course, after we have felt that the idealized models in question have some utility in our generalized goals of strategic idealization of model construction and selection. What is the number of models that are involved that make scientists comfortable? Probably differs between scientists.

    S&O miss the point with the first quote above- it is the idealized commonality, the common ‘truth’ in the midst of the idealized ‘lies’ if you will, that points to the robust hypothesis.

    As to the second quote, robust analysis works because the models have independently idealized different factors in the phenomena, and it follows that when different factors are assumed away, but the outcome is the same, that the common outcome is likely (not proven) to be from the commonality among the models.

  4. I agree with Todd. If we could categorize different types of models so that they don’t “lie” in the same way, and if each type of model leads to similar results, then there is a high probability that the theorem abstracting from the results is correct.

    Of course, as Jenny said, the number of possible models may be very large. We do have the problem of how to differentiate types of models to let them not “lie” in the same way, but it doesn’t mean that robustness is useless.

    As for whether models prove a theorem, I agree with Jake and Todd, that the proof is not conclusive. It’s only about probability. And if we could solve the problem about categorizing different types of models, then the probability should not be as low as 0.001%.

  5. Re Todd’s first paragraph: I agree entirely that whether or not the model is “true” is not the most important issue if it’s even an issue at all. If we’re talking about a theory that isn’t at the fundamental level and you accept reductionism (as I do), then no theory is true because we’re not really sure what happens at the most fundamental level yet. What matters most is predictive power and robustness analysis is good at that because like others have said, it picks out the similarities in models and models rarely “lie” in the same way.

  6. Jenny,

    Surely you don’t think I’m in the business of trivially true statements. What I meant was something more like this:

    If models [p], [q], [r], etc. all imply R, then R is either true or a collective artifact of the models. By this, I don’t mean “P implies (Q v ~Q)”, but that there are no other options, so it’s one or the other.

    The more models that imply R, the better the chance that R is true, rather than a collective artifact.

    As to your point regarding how much a new model increases the odds that R is true, it seems plausible at first glance to reject a linear “every new model increases the odds by .001” account. Does it sound right to say that similar models don’t increase the chances R is true as much as dissimilar models?

    Another concern with the incremental approach is that I could construct trivially different models to get my certainty all the way up to 1.00 fairly easily. This makes me suspect that confidence in robust theorems is asymptotic.

    So, what we’re left with is something like this:
    (1)If models [p], [q], [r], etc. all imply R, then R is either true or a collective artifact of the models.
    (2)R is considered more likely to be true as more models implying R are introduced.
    (3)Dissimilar models that imply R give greater weight to the veracity of R than similar models.
    (4)R cannot be known with certainty; certainty is asymptotic.

    How does this sound?

  7. Jake,

    I was under the impression that you were in the business of giving the business… and business was booming.

    I think you’re on track with the general logic of robustness. I don’t know about similar models yielding smaller increases than dissimilar models. It seems like there might be some effect of the “degree of lying” in the model that would be independent of similarity. Suppose I had models [p]-[r] and I’m going to add to the set [s] and [t], both of which “confirm” the robust theorem. It seems to me that if [s] closely tracks real world phenomena but [t] is marginal on that front, then [s] would increase my confidence more than [t] even if [s] is significantly more similar to [p]-[r] than is [t].

    I suppose you’re right if you add a *ceterus paribus* clause though. If [u] is less similar but maintains significant accuracy, then it would probably raise my confidence more than would [s].

  8. I don’t think robustness analysis has anything to do with confirmation so much as seeing how much the properties of a system can vary while still producing the same property. If you get a lot of different models that produce the same property, then you know that the system can vary a lot while still producing the property. The more “robust” a property is, the more likely the target system will be one of those systems that will produce the property. Robustness analysis doesn’t so much confirm a single given theory as it *creates* a theory that should apply in a lot of cases. It’s still an open question of whether it will apply in the given target system. You will still have to get empirical confirmation to see if the system does actually have fulfill the set of assumptions that will produce the robust property.

  9. Jake,

    You wrote, “If models [p], [q], [r], etc. all imply R, then R is either true or a collective artifact of the models. By this, I don’t mean “P implies (Q v ~Q)”, but that there are no other options, so it’s one or the other.” This fact is not trivially true, you argued.

    By saying that the commonality R is either true or a collective artifact of the models, I take it that you are saying that R is held in common by the models either because R is a feature of the world that the models are correctly identifying or there is a common feature of the models that give rise to R but that don’t say anything true of the world (i.e. R is a collective artifact, in your words). If the models pick out R as being a true feature of the world, then R is true. If the models do not pick out R in this way, then R is false.

    Put this way, it does, in fact, seem as though you’re saying that R is either true or false. And while it may be helpful to know why R is false (i.e. because R is a collective artifact of the models), that fact does not seem particularly important when we talk about whether we can know that robust models are telling us anything true about the world.

    One way to argue around my critique would be to say that from your insights, we can form the following conditional: “if R is not a collective artifact then R is true of the world.” So, if we can discount R as a collective artifact, then we can show that R is true.

    However, there is no way (that I know of anyway) to show that R is not a collective artifact of the models. Thus, we can’t prove our conditional to be true. So I’m not sure that knowing why R may be false gets us anywhere.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: