Peer Review of Design for a Darwinian Brain Part 1

Here we will see peer-review in action. This is a review of the paper submitted here... 

http://arxiv.org/abs/1303.7200

Reviewers comments below, and my responses afterwards...  

FIRST/ THANKS +++++++ for the great reviewers. Some wonderful references here. Yes, it is a confusing paper, I've tried to sort it out. Some of the references are very helpful for me. I can't address all your comments in the paper as I'd like due to space limitations, but I address them here, and maybe we can continue our discussion, especially reviewer 2 who gives some amazing references that really clearly express the problems with most computational neuroscience models these days! 

Many thanks, 
Chrisantha

Dear Chrisantha Fernando,

The current reviewing round for your paper

'Design for a Darwinian Brain: Part 1. Philosophy and Neuroscience'

in the conference

'Living Machines 2013'

has ended.


Enclosed please find the reports.

Major Report:

ARGUMENTATION:
The third reviewer was late with their report, so it is included below:

It seems to me that the aims of this paper is to justify the
philosophy that 'open-ended cognition' is dependent on physical symbol systems, and secondly to propose ways that this might be implemented. As such, it addresses the issue of thinking machines and therefore the topic is well-suited to the Living machines conference. The manuscript
however is ill-structured for the needs of the conference and doesn't seem to have had its audience in mind. At the level of individual paragraphs the paper is well-written and interesting. However, I'm not sure who it was written for and surely it requires a radical simplification in order to present these ideas to the likely audience many without a cognitive philosophy background). I think the topic would be a valuable addition to the conference, however I fear the
paper and any talk based on the current version of the paper, would be unnecessarily complex.

Specific Points
- I detect 3 related "conversations" going on in this paper. (i) the analogy between PSS and chemistry; (ii) the relationship between cognition and evolution; and (iii) how to implement PSSs. The third is surely the topic that fits the conference (and your title) and the paper needs to be structured to best present this (and therefore to introduce Part 2)
- Given the likely audience the paper surely needs to start with a resume of PSSs and the claim that they are essential for the production of open-ended cognition. This would then set-up the aims for the paper (currently lacking) 

Done

- Referencing is occasionally sloppy
- On a related note, I don't mind the conversational style but it has to be all or nothing. One can't state truths in this style, rather one must 'suggest' 

Removed dogmatics 

- p4 who is Marcus and why do they deserve a wole section based on their ideas about a single experiment.

See reference list for an answer to "who is Marcus". His importance is clarified in greater detail in the paper. 

This hardly presents the compelling case for the necessity of PSSs for cognition.

I believe Marcus' book "The Algebraic Mind" does present a compelling case, as does Fodor and Pylyshyn's original paper. 

- p6 end of para 2 "the details ... ...are beyond the scope of this paper" The paper is called "Design for a darwinian brain" therefore we can handle some details surely.

 Details provided. 

Reports:

---------------

ARGUMENTATION:

Suitability for conference?
Ok.

Novelty: intermediate.

Clarity: good.

Overall evaluation? Positive.

Detailed comments (short summary, then advice for author)

In this paper Fernando proposes a wide presentation and discussion about hypothesized computational principles underlying neural mechanisms for open-ended cognition. The argumentation mainly focuses on the notion of physical symbol systems, grounded on both philosophical and computational neuroscience elements.

I think the first part of the paper, which presents philosophical arguments in favor of considering the brain as a physical symbol system, lacks a clear statement of what is the novelty of the proposal and how it helps solving a debate or an open question in the philosophical community. The proposal is interesting.

The novelty of the proposal is that the brain contains data-structures that can be thought of as a physical symbol system being modified by stochastic population based search, with each atom in this physical system being a neuronal modules with neuronal functionality. The inputs and outputs to neuronal modules are evolved when a module undergoes duplication and divergence on the cortical surface. This novelty is emphasised with a diagram. 

But if the paper aims to contribute also at the level of the philosophical debate, the reader needs to be clearly explained: whether the physical symbol system hypothesis (PSSH) is currently debated in philosophy; what are the alternative propositions; in which aspect the present proposal helps go further the original PSSH and answer criticisms.
Please cite Newell and Simon in the first paragraph when addressing the PSSH.

The PSS was criticised for lacking in grounding, i.e. being able to explain how non-symbolic processes in the brain could be coupled with symbolic processes. This is precisely what is explained by our model in Part 2. But this must be clarified in Part 1, and now it is... 

Moreover, some parts of the introduction need to be reformulated so that they do sound as strong as a proclamation of the one and only truth. For instance, it is written that “The genetic regulatory system of the cell is a physical symbol system…”. Please replace “is” (which is not true) by “can be viewed as” or “can be modeled as”. Same thing for the next sentence: “it is a chemical symbol system to be exact”. Page 2, it is written that “combinatorial semantics is then what determines how a molecule with a particular structure will react (behave)”. No, combinatorial semantics is how we, humans, describe how a molecule with a particular structure reacts. Again page 2, “the fact that molecules are chemical symbols …”. One cannot scientifically affirm that they ARE chemical symbols. One can say that they can be described as chemical symbols.

Done

Providing more citations would also be useful to understand which authors are defending which arguments. Page 1, please cite a paper when saying that “the theory of Darwinian neurodynamics proposes […] mechanisms by which these symbol structures can replicate in the brain […] action grammar.” Please also cite a paper for “This has also been the approach of molecular biology…”. Page 2, please cite one or several paper in the sentence “philosophers who argue about representations”.

In general, in this first part of the paper, I think it would be more interesting for the reader to be able to understand where is the debate, why people disagree, why the author chose one particular position rather than another, and how the author’s proposal introduces an add-on/a novel aspect, to the discussion. Otherwise, it may sound too much as a repetition of an old philosophical theory without bringing much novelty.

Done 

The following pages, enumerating and explaining properties of the system, are more structured and information-rich. One question concerning the two plausible candidates for a neuronal symbol system (spike patterns vs connectivity patterns; Page 1): are they mutually exclusive? Please explain why yes or why not.

No they are not. This is elaborated upon. 

Concerning the discussion on whether connectionist models without explicit symbols nor rules could be sufficient for open-ended cognition (pages 3-5), I think it would be useful to discuss the following papers: Rougier NP, Noelle DC, Braver TS, Cohen JD, O’Reilly RC (2005) in Proc Nat Acad Science USA; Tani J, Nolfi S (1999) in Neural Networks. The two papers propose models where complex representations and task resolution can emerge without explicit symbols. In what aspect is this type of approach’s ability to progressively reach efficient modeling of open-ended cognition not convincing? What are the limitations according to the author?

The Rougier paper here correctly criticises itself and gives a nod (most likely due to pressure from a reviewer) to the need for physical symbol systems in its Supplementary Material here, which neatly explains the problem domain that we are interested which extends far beyond WCST, see below... 

"Our primary focus in this work has been on one of the factors that contributes to the flexibility of cognitive

control: the ability to generalize task performance to novel environments, and to switch task representations

dynamically in response to changing demands (as in the WCST task). However, there is another equally

important factor in the flexibility of control: the ability to generate entirely new behaviors. Our model does

not directly address this capacity for generativity. Although the model was tested on its ability to perform

tasks on new combinations of features, it was never asked to perform entirely new tasks. One might contend

that the same is true for humans; the flexibility of cognitive control has its limits. People cannot actually

perform entirely new tasks without training, at least not with any alacrity (e.g., one cannot play complex

card games such as bridge very well when first learning them). Nevertheless, people can recombine familiar

behaviors in novel ways. Such recombination, when built upon a rich vocabulary of primitives, may support

a broad range of behavior and thus contribute to generatively."

It is this generative complexity that PSS are capable of by definition. The paper acknowledges the simplicity of the algorithm implemented, i.e. "random sampling + delayed replacement" and acknowledges the need for mechanisms that can undertake more sophisticated forms of search. 

"Generativity also highlights the centrality of search processes to find and activate the appropriate

combination of representations for a given task, a point that has long been recognized by symbolic

models of cognition (20). Such models have explored a range of search algorithms and heuristics that vary

in complexity and power (e.g., grid search, hill-climbing, means-ends, etc.). Our model implemented a relatively

simple form of search based on stabilizing and destabilizing PFC representations as a function of task

performance, which amounts to random sampling with delayed replacement. This appeared to be sufficient

for the performance of the simple tasks addressed, consistent with previous results (10, 21). An important

focus of future research will be to identify neural mechanisms that implement more sophisticated forms of

search. This is likely to be necessary to account for human performance in more complex problem solving

domains, in which the PFC plays a critical role (22)."

Many thanks for pointing us to this self-criticism, which has been pointed to now in the paper. 
Another issue the above paper brought up is that Rougier et al and apparently others consider 'arbitrary variable binding' to be biologically implausible in the brain. Lets examine this a bit more... 

The Tani and Nolfi paper was much more interesting for me. It determines the inputs to a higher order predictor on the basis of minimising of prediction error of the expert predictors at the lower level. The higher level then predicts the gate opening dynamics of the lower level predictors and sends this upwards. Two levels are modelled. However, the system is severely theoretically limited in contrast to the one proposed in our architecture. The reason is that for a high dimensional space of models, there would be a combinatorial explosion of possible higher-order models which could be composed from combinations of lower order models. This problem has been addressed by Goran Gordon in a set of papers about hierarchical curiosity loops (which are now referenced). An efficient stochastic method for doing parse search in the space of all possible higher order models is needed because as the number of levels increases the combinatorial possibilities of higher order models grows exponentially. For efficient sparse search in higher level models, each expert must be specified by a representation of its specific inputs and outputs. And when new models are being produced at a certain level, they must be made as variants of existing models. Each model effectively becomes a symbolic atom, with its inputs and outputs specifying its connections to other atoms, i.e. permitting arbitrary variable binding. One of the interesting things is that prediction error may be used as a competition function. This would require that there is also some competition based on similarity otherwise the easiest thing to predict would take over the representation of lower level modules. Some things are hard to predict, but you still want to try to predict them, so prediction error itself can't quite be the right selection function eventually. 

Therefore, neither of these two papers propose how complex representations can emerge without explicit symbols. The former because of reasons the authors admit themselves, and the later due to its non-extensibility to high dimensional systems. 

Concerning the discussion of the dichotomy between direct rational thinking such as inference used during human problem solving and incremental blind learning such as natural selection (Pages 7-9). 

It would be wrong to talk derivatively about the blind. Blind is not equal to stupid. 

I think it would be very interesting to discuss the reinforcement learning (RL) framework which has already been extended to include this dichotomy. Researches have formalized the distinction between model-free RL (such as TD-learning) and model-based RL (which can perform inference, planning, and similar computations to the cognitive map mentioned page 9). Moreover, formal mappings between brain structures and components of a computational architecture combining model-based and model-free RL have been proposed (Daw ND, Niv Y, Dayan P (2005) Nat Neurosci; Khamassi M, Humphries MD (2012) Frontiers in Behavioral Neuroscience).

Model based RL is absolutely essential I agree. TD methods are one plausible way to implement delayed reward distribution. However, what DN (Darwinian neurodynamics) proposes is that the representations used by model based RL (the models) can't be produced effectively in an open-ended fashion without being replicated with variation. That is, DN proposes how the space of DN models can be searched over. DN is used for function approximation in high-D spaces that may or may not be combined with TD methods. The options framework has started to do this, but there is NO description elsewhere of how options can be searched in high-D option spaces. DN solves this problem. We have described elsewhere neuronally plausible mechanisms of replicating connectivity patterns which are the proposed substrate for such neuronal 'molecules'/symbol structures. 

As noticed by the author, the RL framework has the advantage of having also been extended to include compositionality and systematicity (with options; Page 9).

But not discovery of novel options on the basis of intrinsic reward functions.

Another powerful aspect of this framework is that it has also been extended to meta-learning mechanisms (learning how to learn) which appear to constitute a key mechanism required for human cognitive control abilities (Khamassi M, Wilson CRE, Rothe M, Quilodran R, Dominey PF, Procyk E (2011) book chapter in Oxford Univ Press; Khamassi M, Enel P, Quilodran R, Dominey PF, Procyk E (2013) Prog Brain Res). 

I agree that meta-learning constitutes a key mechanism required for human cognitive control abilities (if by that you mean the ability to learn effectively by structuring ones exploration distribution). We have explained elsewhere how natural selection can exhibit meta-learning, e.g. by covariance matrix adaptation, and by non-trivial neutrality, and even more powerful learning when combined with Hebbian learning to bias exploration distributions. 

http://www.isir.upmc.fr/files/2011COS1838.pdf

The Tuning of meta-parameters is a task to which the architecture proposed in Part 2 is ideally suited, e.g. a SHC or RL algorithm can be implemented in an actor atom, and once this replicates, the meta-parameters encoded by the atom can be mutated. This meta-learning is called the 'evolution of evolvability' and is discussed in the paper. Modifying learning rates (lowering them when at a local optimum) can sometimes be harmful because it may cause one to get stuck. This is a finding of Clune et al in PLoS Computational Biology, so simple progress monitoring methods can sometimes be sub-optimal. Evolutionary biology has a large literature on meta-learning that is in many ways more sophisticated than that of Doya. 

In general, I find that the author could more explicitly stress an idea that beautifully arises from her previous work: that genetic algorithms could be seen as an alternative to model brain learning mechanisms, which are classically modeled by RL mechanisms.

Lets be clear, this is not an idea that beautifully arises from her previous work, it IS 'her' previous work :) 

Both approaches are optimization techniques. Each one has its advantages and limitations. Investigating both in parallel could lead to different conclusions, or at least could complete each other by explaining things that are not well explained by the alternative approach. I think the paper would greatly benefit from discussing this further.

We will point to this interaction 

Small typos:
Bottom of page 5, axonel -> axonal
Bottom of page 6, commends -> commands
Page 8, briefly first -> briefly discussed first
Page 9, systemeticity -> systematicity


Are there any typographical or layout issues? (An extended abstract has a maximum of 3 pages max and a full paper has a maximum of 12 pages; papers should be in the standard Springer LNCS format.)

No.



---------------