Methodology for the Social Sciences

02 Jan 2024 13:40

---

Nearly every sociological thesis proposes a new method, which, however, its author is very careful not to apply, so that sociology is the science with the greatest number of methods and the least results. --- Henri Poincaré, Science and Method, chapter I [*]

That is: what are the appropriate methods for studying social or cultural phenomena in a scientific way? In principle, this is a sub-division of general scientific methodology, but arguably (this is one of the big questions here!) social phenomena are sufficiently different from natural ones that they need truly distinctive methods. (Or perhaps social phenomena can be studied with the same methods as biological ones, but both are distinctive from inorganic nature.) It seems to be true that how one should study society depends on what society is like, i.e., general issues of social theory. But my hope is to learn something about methods which are relatively agnostic about social ontology, because they'd work even under very different assumptuions about the nature of society.

It's probably a bad thing that so many of my favorite works in this genre are relentlessly negative.

*: An even better line, sometimes attributed to Poincaré, is "Sociolgists discuss sociological methods; physicists discuss physics" (e.g., Christopher Alexander, Notes on the Synthesis of Form, preface [Harvard University Press, revised edition, 1971]). But I can't locate this in Poincaré's works (which, admittedly, I can only read in translation), and suspect it is the product of someone's memory working on, and improving, what I did quote, especially since that comes at the end of a paragraph which also talks about methods in physics. ^

See also: Agent-Based Modeling; Archaeology; Economics; Historical Materialism; Historiography; Mechanistic Explanations; Network Data Analysis; Scientific Method and Philosophy of Science; Sociology; Statistics

Recommended whole-heartedly:
• Stanislav Andreski
  • Elements of Comparative Sociology
  • Social Science as Sorcery
• Scott Ashworth, Christopher R. Berry and Ehtan Bueno de Mesquita, Theory and Credibility: Integrating Theoretical and Empirical Social Science
• Jon Elster
  • "Excessive Ambitions", Capitalism and Society 4:2 (2009): 1
  • "Marxism, Functionalism and Game Theory: The Case for Methodological Individualism," Theory and Society 11 (1982): 453--482 [JSTOR]
  • Nuts and Bolts for the Social Sciences
  • "A Plea for Mechanisms", in Hedstrom and Swedberg (eds.), Social Mechanisms: An Analytical Approach to Social Theory
• Kieran Healy
  • "Fuck Nuance" ["Abstract: Seriously, fuck it."]
  • "A Comprehensive Guide to Social Scientists"
• Peter Hedstrom, Dissecting the Social: On the Principles of Analytical Sociology
• ibn Khaldûn
• Charles E. Lindblom and David K. Cohen, Usable Knowledge: Social Science and Social Problem Solving [Why social science will almost never be able to acheive the kind of rational authority that the natural sciences possess, and some suggestions about how social scientists might instead direct their efforts so as to be useful in solving social problems.]
• Karl R. Popper, The Poverty of Historicism
• Alexander Rosenberg, Economics: Mathematical Politics or Science of Diminishing Returns?
• W. G. Runciman, A Treatise on Social Theory [This is a trilogy, of which I've finished the first, methodological volume...]
• John R. Sutton, Marshall's Tendencies: What Economists Can Know [An exposition of the strengths and limits of the usual econometric approach (statistical inference within a fully-specified model), and Sutton's alternative, of deriving results, usually inequalities, which hold uniformly across broad ranges of models.]
• Charles Tilly
  • Many of Tilly's methodological papers are available online
  • Big Structures, Large Processes, Huge Comparisons
  • Explaining Social Processes
• Tal Yarkoni, "The Generalizability Crisis", Behavioral and Brain Sciences forthcoming (2020+), psyarxiv/jqw35 [While focused on psychology, many of the issues apply with very little change to other social sciences]

Recommended whole-heartedly, close-ups:
• >Samuel J. Gershman and Tomer D. Ullman, "Causal implicatures from correlational statements", PLoS ONE 18 (2023): e0286067 [Comments under Causal Inference]
• Kevin D. Hoover, "Reductionism in Economics: Intentionality and Eschatological Justification in the Microfoundations of Macroeconomics" [PDF preprint via Prof. Hoover]
• Morgan Kelly, "The Standard Errors of Persistence" [PDF preprint, 2019]
• Gary King and Margaret Roberts, "How Robust Standard Errors Expose Methodological Problems They Do Not Fix" [PDF preprint]
• Youjin Lee and Elizabeth L. Ogburn
  • "Testing for Network and Spatial Autocorrelation", arxiv:1710.03296
  • "Network Dependence and Confounding by Network Structure Lead to Invalid Inference", arxiv:1908.00520
• Momin M. Malik, Bias and beyond in digital trace data [Ph.D. thesis, 2018, CMU Institute for Software Research]
• Paul E. Meehl
  • "Why Summaries of Research on Psychological Theories Are Often Uninterpretable", Psychological Reports 66 (1990): 195--244 [PDF reprint]
  • "Theory-Testing in Psychology and Physics: A Methodological Paradox", Philosophy of Science 34 (1967): 103--115 [PDF reprint]
• Brendan O'Connor, Statistical Text Analysis for Social Science [Ph.D. thesis, 2014, CMU department of Machine Learning]
• Thomas B. Pepinsky, "On Whorfian Socioeconomics", SSRN/33123347
• Ole Rogeberg and Hans Olav Melberg, "Acceptance of unsupported claims about reality: a blind spot in economics", Journal of Economic Methodology 18 (2011): 29--52 [By no means is the problem described here limited to economics, though economists may be unusually blind to it, owing to the entirely malign and unwarranted influence of Milton Friedman. See also]
• Dan Sperber, "A naturalistic ontology for mechanistic explanations in the social sciences" [online]
• Christopher Tosh, Philip Greengard, Ben Goodrich, Andrew Gelman, Aki Vehtari, and Daniel Hsu, "The piranha problem: Large effects swimming in a small pond", arxiv:2105.13445 [On the impossibility of finding lots of variables having large effects on the same target variable, without those variables also having large effects on each other]
• Michael D. Ward, Brian D. Greenhill and Kristin M. Bakke, "The perils of policy by p-value: Predicting civil conflicts", Journal of Peace Research 47 (2010): 363--375
• Jake Westfall and Tal Yarkoni, "Statistically controlling for confounding constructs is harder than you think", PLoS ONE 11 (2016): e0152719

Recommended half-heartedly:
• Stanley Lieberson and Freda B. Lynn, "Barking Up the Wrong Branch: Scientific Alternatives to the Current Model Sociological Science," Annual Review of Sociology 28 (2002): 1--19 [I'm sympathetic, but would offer the correction that even what sociologists try to do isn't like classical physics at all, though it may be like what they imagine physics to be. Also, a lot of this is very similar to what Popper said in The Poverty of Historicism. PDF reprint via Prof. Lieberson.]
• Stanley Lieberson and Joel Horwich, "Implication Analysis: A Pragmatic Proposal for Linking Theory and Data in the Social Sciences", Sociological Metholodgy 38 (2008): 1--50, with discussions and replies, pp. 51--100 [PDF via Prof. Lieberson. These are all reasonable bits of advice, but I alternated between thinking "This needs saying?!?" and "Sadly, this needs saying". The best short summary is provided by Mizruchi on p. 68; this makes it clear that what Lieberson and Horwich propose is far too shapeless to count as a method of analysis. (Unfortunately Mizruchi goes on to rather spoil the effect by showing [p. 70] that either he has no conception of what it means to compare alternative explanations, or he doesn't understand what multiple linear regression does, or perhaps both.) One of the shrewdest comments is that by Tilly: this is obviously the right way to go, and it will involve a substantial change in how sociologists reproduce themselves professionally.]
• Scott Moss and Bruce Edmonds, "Sociology and Simulation: Statistical and Qualitative Cross‐Validation", American Journal of Sociology 110 (2005): 1095--1131 [Comments]

To read, historical, primary sources:
• James M. Beshers (ed.), Computer Methods in the Analysis of Large-Scale Social Systems [Proceedings of a 1964 conference]
• Arthur Stinchcombe, Constructing Social Theories

To read, historical, secondary sources:
• Paul Erickson, Judy L. Klein, Lorraine Daston, Rebecca Lemov, Thomas Sturm and Michael D. Gordin, How Reason Almost Lost Its Mind: The Strange Career of Cold War Rationality

To read:
• Andrew Abbott
  • "Transcending General Linear Reality", Sociological Theory 6 (1988): 169--186
  • Chaos of Disciplines
  • Methods of Discovery: Heuristics for the Social Sciences
• Eckhart Arnold, "Tools or Toys? On Specific Challenges for Modeling and the Epistemology of Models and Computer Simulations in the Social Sciences", phil-sci/5424
• Derek Beach and Rasmus Brun Pedersen, Process-Tracing Methods: Foundations and Guidelines
• Andrew Bennett, "Process Tracing and Causal Inference", phil-sci/8872
• Richard Biernack, Reinventing Evidence in Social Inquiry: Decoding Facts and Variables
• Raymond Boudon
  • "Beyond Rational Choice Theory", Annual Review of Sociology 29 (2003): 1--21
  • The Crisis in Sociology
  • The Logic of Sociological Explanation
• Ian Bradley and Ronald L. Meek, Matrices and Society: Matrix Algebra and Its Applications in the Social Sciences
• Elizabeth Bruch and Jon Atwell, "Agent-Based Models in Empirical Social Research", Sociological Methods & Research 44 (2015): 186--221
• James S. Coleman, The Foundations of Social Theory
• Pierre Demeulenaere (ed.), Analytical sociology and social mechanisms
• Jared Diamond and James A. Robinson (eds.), Natural Experiments of History
• E. Anthon Eff, "Does Mr. Galton Still Have a Problem? Autocorrelation in the Standard Cross-Cultural Sample", World Cultures 15 (2004): 153--170 [PDF]
• Jon Elster, Explaining Social Behavior: More Nuts and Bolts for the Social Sciences [2nd edition of Nuts and Bolts, much-expanded; much less rational-choicy]
• Glenn Firebaugh, Seven Rules for Social Research
• Roberto Franzosi, From Words to Numbers: A Journey in the Methodology of the Social Sciences [I heard Franzosi talk about this at the quantitative methodology seminar at Ann Arbor, Sept. 2004, and was very impressed; I have since read the first half and remain very impressed.]
• Neil Gross, "Charles Tilly and American Pragmatism", The American Sociologist 41 (2010): 337--357
• John R. Hall, Cultures of Inquiry: From Epistemology to Discourse in Sociohistorical Research
• Eszter Hargittai (ed.), Research Confidential: Solutions to Problems Most Social Scientists Pretend They Never Have
• Geoffrey Hawthorn, Plausible Worlds: Possibility and Understanding in History and the Social Sciences
• Jeffrey Haydu, "Reversals of fortune: path dependency, problem solving, and temporal cases", Theory and Society 39 (2010): 25--48
• Peter Hedström an Peter Bearman (eds.), The Oxford Handbook of Analytical Sociology
• Peter Hedström and Gianluca Manzo (eds.), Agent-Based Modeling: Advances and Challenges, special issue of Sociological Methods and Research 44:2 (2015) [editors' introduction]
• David K. Henderson, Interpretation and Explanation in the Human Sciences
• Donald W. Katzner
  • Models, Mathematics, and Methodology in Economic Explanation
  • Analysis without Measurement
• Gary King et al., Designing Social Inquiry
• John I. Kitsue and Aaron V. Cicourel, "A Note on the Uses of Official Statistics", Social Problems 11 (1963): 131--139
• Dean Knox, Christopher Lucas, and Wendy K. Tam Cho, "Testing Causal Theories with Learned Proxies", Annual Review of Political Science 25 (2022): forthcoming
• Erin Leahey, "Methodological Memes and Mores: Toward a Sociology of Social Research", Annual Review of Sociology 34 (2008): 33--53
• Stanley Lieberson, Making It Count: The Improvement of Social Research and Theory
• Charles Lindblom, Inquiry and Change: The Troubled Attempt to Understand and Shape Society
• Kristin Luker, Salsa Dancing into the Social Sciences: Research in an Age of Info-glut
• John Levi Martin
  • The Explanation of Social Action
  • Thinking Through Methods: A Social Science Primer
  • Thinking Through Statistics
  • Thinking Through Theory
• R. K. Merton, Social Theory and Social Structure
• John W. Mohr, Christopher A. Bail, Margaret Frye, Jennifer C. Lena, Omar Lizardo, Terence E. McDonnell, Ann Mische, Iddo Tavory and Frederick F. Wherry, Measuring Culture
• Diana C. Mutz, Population-Based Survey Experiments
• D. C. Phillips, Holistic Thought in Social Science
• Amy R. Poteete, Marco A. Janssen and Elinor Ostrom, Working Together: Collective Action, the Commons, and Multiple Methods in Practice
• Charles C. Ragin, The Comparative Method: Moving Beyond Qualtiative and Quantitative Strategies
• Dietrich Rueschemeyer, Usable Theory: Analytic Tools for Social and Political Research
• William H. Sewell Jr., Logics of History: Social Theory and Social Transformation
• Richard Swedberg
  • The Art of Social Theory
  • "Theorizing in sociology and social science: turning to the context of discovery", Theory and Society 41 (2012): 1--40
• Iddo Tavory and Stefan Timmermans, Abductive Analysis: Theorizing Qualitative Research
• Charles Tilly
  • "Mechanisms and Political Processes", Annual Review of Political Science 4 (2001)
  • "Micro, Macro, or Megrim?" [Which I somehow have as a PDF preprint, "Columbia University, August 1997"]
• Kenneth I. Wolpin, The Limits of Inference without Theory