Moving beyond content‐specific computation in artificial neural networks

Nicholas Shea

doi:10.1111/mila.12387

Abstract

A basic deep neural network (DNN) is trained to exhibit a large set of input–output dispositions. While being a good model of the way humans perform some tasks automatically, without deliberative reasoning, more is needed to approach human‐like artificial intelligence. Analysing recent additions brings to light a distinction between two fundamentally different styles of computation: content‐specific and non‐content‐specific computation (as first defined here). For example, deep episodic RL networks draw on both. So does human conceptual reasoning. Combining the two takes advantage of the complementary costs and benefits of each. It also offers a better model of human cognitive competence.

Highlights

For those of us who have long championed artificial neural networks (ANNs), 2012 was a good year
Adding a store of explicit memories has overcome some of the limitations of the basic deep neural networks (DNNs) architecture
We have seen that the ability to perform non‐ content‐specific computations is key

Summary

Introduction

For those of us who have long championed artificial neural networks (ANNs), 2012 was a good year. Special‐purpose computer chips, a variety of technical tricks, and huge databases of training data, neural networks with multiple hidden layers began to out‐compete all other computational systems on several benchmarks (Buckner, 2018). These deep neural networks (DNNs) have demonstrated impressive abilities in one domain after another, including: image classification (Krizhevsky et al, 2012; Eslami et al, 2018), strategic games (computer games: Mnih et al, 2015; Go: Silver et al, 2016), natural language processing (Bahdanau et al, 2014; Brown et al, 2020; Floridi & Chiriatti, 2020) and protein folding (Senior et al, 2020; Jumper et al, 2020 [abstract]). When tasked with learning something new – acquiring an additional input‐output disposition – they are always at risk of catastrophically forgetting what has been so laboriously learnt to date (French, 1999)

Objectives

Findings

Conclusion