Methodological comparison

We compared EPI to two alternative simulation-based inference techniques, since the likelihood of these eigenvalues given 𝐳 is not available. Approximate bayesian computation (ABC) (Beaumont et al., 2002) is a rejection sampling technique for obtaining sets of parameters 𝐳 that produce activity 𝐱 close to some observed data 𝐱0. Sequential Monte Carlo approximate bayesian computation (SMC-ABC) is the state-of-the-art ABC method, which leverages SMC techniques to improve sampling speed. We ran SMC-ABC with the pyABC package (Klinger et al., 2018) to infer RNNs with stable amplification: connectivities having eigenvalues within an ϵ-defined l−2 distance of

SMC-ABC was run with a uniform prior over 𝐳∈[-1,1](4⁢N), a population size of 1000 particles with simulations parallelized over 32 cores, and a multivariate normal transition model.

SNPE, the next approach in our comparison, is far more similar to EPI. Like EPI, SNPE treats parameters in mechanistic models with deep probability distributions, yet the two learning algorithms are categorically different. SNPE uses a two-network architecture to approximate the posterior distribution of the model conditioned on observed data 𝐱0. The amortizing network maps observations 𝐱i to the parameters of the deep probability distribution. The weights and biases of the parameter network are optimized by sequentially augmenting the training data with additional pairs (𝐳i, 𝐱i) based on the most recent posterior approximation. This sequential procedure is important to get training data 𝐳i to be closer to the true posterior, and 𝐱i to be closer to the observed data. For the deep probability distribution architecture, we chose a masked autoregressive flow with affine couplings (the default choice), three transforms, 50 hidden units, and a normalizing flow mapping to the support as in EPI. This architectural choice closely tracked the size of the architecture used by EPI (Figure 2—figure supplement 1). As in SMC-ABC, we ran SNPE with 𝐱0=μ. All SNPE optimizations were run for a limit of 1.5 days, or until two consecutive rounds resulted in a validation log probability lower than the maximum observed for that random seed. It was always faster to run SNPE on a CPU with 32 cores rather than on a Tesla V100 GPU.

To compare the efficiency of these algorithms for inferring RNN connectivity distributions producing stable amplification, we develop a convergence criteria that can be used across methods. While EPI has its own hypothesis testing convergence criteria for the emergent property, it would not make sense to use this criteria on SNPE and SMC-ABC which do not constrain the means and variances of their predictions. Instead, we consider EPI and SNPE to have converged after completing its most recent optimization epoch (EPI) or round (SNPE) in which the distance |𝔼𝐳,𝐱⁢[f⁢(𝐱;𝐳)]-𝝁|2 is less than 0.5. We consider SMC-ABC to have converged once the population produces samples within the ϵ=0.5 ball ensuring stable amplification.

When assessing the scalability of SNPE, it is important to check that alternative hyperparamterizations could not yield better performance. Key hyperparameters of the SNPE optimization are the number of simulations per round nround, the number of atoms used in the atomic proposals of the SNPE-C algorithm (Greenberg, 2019), and the batch size n. To match EPI, we used a batch size of n=200 for N≤25, however we found n=1,000 to be helpful for SNPE in higher dimensions. While nround=1,000 yielded SNPE convergence for N≤25, we found that a substantial increase to nround=25,000 yielded more consistent convergence at N=50 (Figure 2—figure supplement 2A). By increasing nround, we also necessarily increase the duration of each round. At N=100, we tried two hyperparameter modifications. As suggested in Greenberg, 2019, we increased natom by an order of magnitude to improve gradient quality, but this had little effect on the optimization (much overlap between same random seeds) (Figure 2—figure supplement 2B). Finally, we increased nround by an order of magnitude, which yielded convergence in one case, but no others. We found no way to improve the convergence rate of SNPE without making more aggressive hyperparameter choices requiring high numbers of simulations. In Figure 2C–D, we show samples from the random seed resulting in emergent property convergence at greatest entropy (EPI), the random seed resulting in greatest validation log probability (SNPE), and the result of all converged random seeds (SMC).