A cognitive architecture (CA) is a unified theory of cognition which describes how sub-systems such as perception, motor action, and memory interact and function as a coherent system to produce a wide variety of human behaviors, ranging from simple behaviors to complex, goal-oriented behaviors (e.g., Anderson, 2007, Newell, 1990). Unlike a theory of a specific sub-system, such as visual perception, the scope of a CA is more comprehensive, striving to integrate findings across many domains and identify invariant properties in the structure and function of cognition. Newell (1990) argued that CAs are useful scientific tools for preventing myopia and fragmentation of theory in cognitive science. His call for action served as a catalyst for many years of research guided by CAs.

During the intervening time, a total of 84 CAs have been proposed, including 49 still under active development (Kotseruba & Tsotsos, 2020). CAs are based on a wide variety of assumptions and theoretical motivations, which fall into the following broad categories: symbolic, hybrid symbolic/sub-symbolic, and neurally-based architectures. Symbolic architectures, such as EPIC (Kieras & Meyer, 1997), represent knowledge as abstract symbols on which rules operate to produce higher-order cognition, such as reasoning. Hybrid CAs, such as ACT-R (Anderson, Bothell, Byrne, Douglass, Lebiere et al., 2004), incorporate elements of symbolic architectures and sub-symbolic architectures. Other CAs, such as SPA (Eliasmith & Anderson, 2003), place more emphasis on describing the neural systems underlying cognition and learning, and may vary in biological parity.

Several interesting questions arise in light of the plethora of CAs: why are there so many CAs, and how should they be analyzed and compared? Regarding the first question, the large number of CAs, to some extent, reflects a wide diversity of ideas for developing an integrated theory of cognition, which can also be specified at different levels of analysis. However, we also believe the large number of CAs stems from the lack of common framework for analyzing and comparing CAs, making it difficult to converge on a smaller set of viable candidates. This leads us to the second question: how should CAs be evaluated and compared? Previously, the Newell Test has been proposed as a method for testing CAs Anderson and Lebiere, 2003, Newell, 1990. The Newell Test is a minimal set of empirical phenomena and cognitive abilities a CA must satisfy to be considered viable. Although the Newell Test is useful for ensuring that a CA accounts for a wide range of empirical phenomena, it suffers from three limitations: (1) it cannot distinguish between multiple CAs which satisfy the test, (2) it does not directly test the core theory of a CA, and (3) it lacks a formal method for distinguishing between information processing characteristics, such as serial and parallel processes. In light of these limitations, we believe the Newell Test should be supplemented with other approaches, such as the one we present below.

One salient challenge in testing CAs is disentangling auxiliary assumptions from core architectural assumptions. A core architectural assumption refers to an invariant property of mental representation, cognitive functioning, or interaction between sub-systems that is central to the theory underlying the CA. By contrast, an auxiliary assumption refers non-critical aspect of a CA which can be changed, such as parametric and distributional assumptions. Theoretical progress occurs when empirical tests are designed to test core architectural assumptions, rather than auxiliary assumptions. Without the ability to disentangle auxiliary and core architectural assumptions, experiments provide little basis for distinguishing between CAs. In addition, this situation makes it difficult to understand the performance of CA on the Newell Test (e.g., is performance robust to changes in auxiliary assumptions?). Taken together, this creates a type of “credit assignment” problem where it is often unclear whether a prediction of a CA is due to auxiliary or core architectural assumptions, or a combination thereof (e.g., Duhem, 1954, Kellen, Davis-Stober, et al., 2021).

Distinguishing between auxiliary and core architectural assumptions is challenging for two reasons. First, compared to other modeling approaches, mathematical analysis of CAs is more difficult due to their sheer size and scope. Second, the hierarchical structuring of assumptions in many CAs leads to larger prediction spaces and thus greater flexibility. In isolation, an assumption regarding a sub-system might be considered a core assumption; however, the sub-system itself might be considered an auxiliary assumption at the level of the architecture. For example, numerous variations of ACT-R’s visual sub-system have been proposed (e.g., Nyamsuren and Taatgen, 2013, Salvucci, 2001), meaning results from one particular variation does not provide a clear test of the architecture. Any visual system can be substituted into the architecture, provided that it does not conflict with any core assumptions.

The fundamental question in which we are interested is the following: what constraints does a CA impose on its prediction space? In other words, when an architecture is stripped of auxiliary assumptions (e.g., distributional form, sub-systems, and other modifiable constructs), what behaviors does it produce — and more importantly — what behaviors does it preclude? If a CA provides no constraints on the prediction space, then it is merely an unfalsifiable, assortment of sub-systems arbitrarily combined together.1 By contrast, if a CA does impose constraints, there might be opportunities for developing critical tests (e.g., Birnbaum, 2011). What is needed is a framework for deriving theorems from core architectural assumptions and formulating critical tests.

To fulfill this need, we implemented a global model analysis (GMA; e.g., Pitt, Kim, Navarro, & Myung, 2006) approach leveraging systems factorial technology (SFT; Little et al., 2017, Townsend and Nozawa, 1995), resulting in a framework which we call SFT-GMA. SFT is a mathematical framework and experimental methodology for analyzing the information processing characteristics of a system along four dimensions—architecture, stopping rule, stochastic dependence, and workload capacity (Townsend & Nozawa, 1995). GMA analyzes the entire prediction space of a model to identify testable constraints. SFT can be conceptualized as specific instance of GMA, but this relationship is often implicit in descriptions and applications of SFT. Our rationale for making the role of GMA in SFT explicit is to provide unfamiliar researchers with a systematic framework for testing core architectural assumptions using SFT. In the integrated SFT-GMA framework, architecture, stopping rule, stochastic dependence, and workload capacity form the basis for a SFT model space, and GMA is used to identify constraints the CA imposes on the SFT model space. Using SFT-GMA to analyze and test core architectural assumptions of CAs has three advantages: (1) the SFT model space is based on broad classes of non-parametric models which reduces reliance on auxiliary assumptions, (2) as with GMA methods in general, it can inform the design of critical tests to distinguish between CAs empirically (e.g., Pitt et al., 2006), and (3) SFT provides established and rigorous methods for carrying out empirical tests.