ISLE 5 Conference Workshops
1. The “quantitative crisis”, cumulative science, and English linguistics
Lukas Sönning & Valentin Werner (University of Bamberg, Germany)
Timetable
|
 |
|||||||||||||||||||||||||||||||||||||||
 |
 |
|||||||||||||||||||||||||||||||||||||||
All timings are approximate.
Summary
Scientific progress is a cumulative process of uncertainty reduction that can only succeed if science itself remains the greatest skeptic of its explanatory claims. (Open Science Collaboration 2015: 7)
This workshop seeks to situate (English) linguistics within current developments in the scientific world at large. Recently, fundamental issues in scientific reasoning have been vividly debated in a number of scholarly disciplines with strong quantitative traditions (Ioannidis 2005; see Open Science Collaboration & Kappes 2014 for a summary overview). Relevant publications – referring to a paradigm labelled the “quantitative crisis” – have explored major shortcomings of current research and publication practice, which may inhibit the ultimate goal of scholarly study: progress in knowledge construction (or uncertainty reduction). It is the explicit aim of these publications to make researchers aware of issues relating, for instance, to study design, adequate use of statistical methods, overreliance on p-values, selective reporting of results, etc.
Among the focal problems identified within this broader discourse are
- non-reproducibility of studies, as, for example, original data and analysis procedures are not accessible;
- concerns about high rates of false-positive findings in the published scientific literature;
- overreliance on the results of a single “authoritative” study, on which an influential theory or even an entire research paradigm may be based (also induced by (i));
- lack of transparency in methodology and analysis decisions;
- negligence of replication studies as purportedly “unoriginal” (and unprestigious) despite their potential to put previous findings in perspective.
Evidently, (i)–(v) stand in stark contrast to the ultimate goal of scientific study stated above, and may even violate what are viewed as fundamental maxims of scholarly work. Further, from a more practical perspective, (i)–(v) may translate into a systematic, but unwarranted publication bias (Ioannidis et al. 2014) toward the “new” (research gap filling) and the “significant” (positive rather than null or ambiguous findings).
In an attempt to address the aforementioned issues, authors have developed suggestions for overcoming the “quantitative crisis” (see, e.g., Open Science Collaboration & Kappes 2014; Munafò et al. 2017). More specifically, they have made a particularly strong case for the value of cumulative science (involving, e.g., replication and meta-studies) and transparency (involving, e.g., open data and pre-registration of studies to avoid selective reporting of findings) to achieve robust and reliable results. Collaborative initiatives such as the Center for Open Science (www.cos.io) and platforms such as the Open Science Framework (www.osf.io) have been founded to foster these aims among the scientific community.
Linguistics also has developed into a firmly quantitative discipline (see, e.g., Gries 2015), with the implicit claim to adhere to the “scientific method” (Doyle 2005). However, despite repeated calls in the literature (e.g. Stubbs 2001; Arppe et al. 2010; Egbert & Baker 2016), and despite efforts in neighboring fields, such as psychology (e.g. Pashler & Harris 2012), second language acquisition research (e.g. Oswald & Plonsky 2010), applied linguistics (e.g. Norris & Ortega 2007), and typology (e.g. Haspelmath & Siegmund 2006), it is striking that a discourse similar to the one presented above has to date not ensued in the area of descriptive linguistics (see Fasold 1982 and Aguilar-Sánchez 2014 for rare exceptions).
Thus, based on the premise that linguists studying English have regularly taken a lead role as methodological innovators (as argued by Mair 2014, for instance), it is the aim of this workshop to provide a forum for debating the implications of the “quantitative crisis” and attempts toward its solution for (English) linguistics. We argue that it is genuinely worth considering whether and how the issues and solutions listed above equally apply to our field, and where discipline-specific adaptations are needed. In this spirit, the workshop welcomes contributions that
- discuss the application of alternative or complementary approaches to data analysis (e.g. Bayesian inference, estimation using effect sizes and confidence intervals, use of regularization techniques);
- discuss and outline the potential and limitations of cumulative knowledge construction in linguistics (e.g. through case studies applying meta-analysis or close or conceptual replication);
- explore how principles of open science have been and could be implemented within the linguistic community (e.g. through the affordances of electronic media);
- discuss repercussions of (i)–(iii) for the linguistic publication system (e.g. open science badges, the review process, reporting guidelines, issues pertaining to the “novelty over reliability” bias);
- discuss repercussions of (i)–(iii) for the training of (future) researchers (e.g. design of textbooks and courses on research methodology, development of statistical literacy).
References
Aguilar-Sánchez, Jorge. 2014. Replicability of (socio)linguistics studies. Journal of Research Design and Statistics in Linguistics and Communication Science 1(1), 5–25.
Arppe Antti, Gaëtanelle Gilquin, Dylan Glynn, Martin Hilpert & Arne Zeschel. 2010. Cognitive corpus linguistics: Five points of debate on current theory and methodology. Corpora 5(1), 1–27.
Doyle, Paul. 2005. Replication and corpus linguistics: Lexical networks in text. Paper presented at Corpus Linguistics. Birmingham, July 2005.
Egbert, Jesse & Paul Baker. 2016. Research synthesis. In Paul Baker & Jesse Egbert (eds.), Triangulating Methodological Approaches in Corpus Linguistic Research. New York: Routledge, 183–207.
Fasold, Ralph W. 1982. The amazing replicability of a sociolinguistic pattern. Research on Language and Social Interaction 15(1), 1–12.
Gries, Stefan Th. 2015. Some current quantitative problems in corpus linguistics and a sketch of some solutions. Language & Linguistics 16(1), 93–117.
Haspelmath, Martin & Sven Siegmund. 2006. Simulating the replication of some of Greenberg’s word order generalizations. Linguistic Typology 10(1), 74–82.
Ioannidis, John P.A. 2005. Why most published research findings are false. PLoS Medicine 2(8), e124.
Ioannidis, John P.A., Marcus R. Munafò, Paolo Fusar-Poli, Brian A. Nosek, Sean P. David. 2014. Publication and other reporting biases in cognitive sciences: Detection, prevalence, and prevention. Trends in Cognitive Sciences 18(5), 235–241.
Mair, Christian. 2014. The "English Language Complex", English language studies, and the linguistics of English: The terrain, the landscape, and the map. Plenary presented at ISLE 3: Building Bridges. Zurich, August 2014.
Munafò, Marcus R., Brian A. Nosek, Dorothy V.M. Bishop, Katherine S. Button, Christopher D. Chambers, Nathalie Percie du Sert, Uri Simonsohn, Eric-Jan Wagenmakers, Jennifer J. Ware & John P.A. Ioannidis. 2017. A manifesto for reproducible research. Nature Human Behaviour 1, 0021.
Norris, John M. & Lourdes Ortega. 2007. The future of research synthesis in applied linguistics: Beyond art or science. TESOL Quarterly 41(4), 805–815.
Open Science Collaboration. 2015. Estimating the reproducibility of psychological science. Science 349 (6251), 1–8.
Open Science Collaboration & Heather Barry Kappes. 2014. The Reproducibility Project: A model of large-scale collaboration for empirical research on reproducibility. In Victoria Stodden, Friedrich Leisch & Roger D. Peng (eds.), Implementing Reproducible Research. New York: CRC Press, 299–324.
Oswald, Frederick L. & Luke Plonsky. 2010. Meta-analysis in second language research: Choices and challenges. Annual Review of Applied Linguistics 30, 85–110.
Pashler, Harold & Christine R. Harris. 2012. Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science 7(6), 531–536.
Stubbs, Michael. 2001. Words and Phrases. Oxford: Wiley-Blackwell.
Email the Organising Committee: isle5.conference@ucl.ac.uk
This page last modified 22 May, 2018 by Survey Web Administrator.