Fast and robust analog in-memory deep neural network training.

Analog in-memory computing is a promising future technology for efficiently accelerating deep learning networks. While using in-memory computing to accelerate the inference phase has been studied extensively, accelerating the training phase has received less attention, despite its arguably much larger compute demand to accelerate. While some analog in-memory training algorithms have been suggested, they either invoke significant amount of auxiliary digital compute-accumulating the gradient in digital floating point precision, limiting the potential speed-up-or suffer from the need for near perfectly programming reference conductance values to establish an algorithmic zero point.

Analog Resistive Switching Devices for Training Deep Neural Networks with the Novel Tiki-Taka Algorithm.

A critical bottleneck for the training of large neural networks (NNs) is communication with off-chip memory. A promising mitigation effort consists of integrating crossbar arrays of analogue memories in the Back-End-Of-Line, to store the NN parameters and efficiently perform the required synaptic operations. The "" algorithm was developed to facilitate NN training in the presence of device nonidealities.

Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based accelerators.

Analog in-memory computing-a promising approach for energy-efficient acceleration of deep learning workloads-computes matrix-vector multiplications but only approximately, due to nonidealities that often are non-deterministic or nonlinear. This can adversely impact the achievable inference accuracy. Here, we develop an hardware-aware retraining approach to systematically examine the accuracy of analog in-memory computing across multiple network topologies, and investigate sensitivity and robustness to a broad set of nonidealities.

Optimised weight programming for analogue memory-based deep neural networks.

Analogue memory-based deep neural networks provide energy-efficiency and per-area throughput gains relative to state-of-the-art digital counterparts such as graphics processing units. Recent advances focus largely on hardware-aware algorithmic training and improvements to circuits, architectures, and memory devices. Optimal translation of software-trained weights into analogue hardware weights-given the plethora of complex memory non-idealities-represents an equally important task.

Toward Software-Equivalent Accuracy on Transformer-Based Deep Neural Networks With Analog Memory Devices.

Recent advances in deep learning have been driven by ever-increasing model sizes, with networks growing to millions or even billions of parameters. Such enormous models call for fast and energy-efficient hardware accelerators. We study the potential of Analog AI accelerators based on Non-Volatile Memory, in particular Phase Change Memory (PCM), for software-equivalent accurate inference of natural language processing applications.

RAPA-ConvNets: Modified Convolutional Networks for Accelerated Training on Architectures With Analog Arrays.

Analog arrays are a promising emerging hardware technology with the potential to drastically speed up deep learning. Their main advantage is that they employ analog circuitry to compute matrix-vector products in constant time, irrespective of the size of the matrix. However, ConvNets map very unfavorably onto analog arrays when done in a straight-forward manner, because kernel matrices are typically small and the constant time operation needs to be sequentially iterated a large number of times.

Autapses enhance bursting and coincidence detection in neocortical pyramidal cells.

Autapses are synaptic contacts of a neuron's axon onto its own dendrite and soma. In the neocortex, self-inhibiting autapses in GABAergic interneurons are abundant in number and play critical roles in regulating spike precision and network activity. Here we examine whether the principal glutamatergic pyramidal cells (PCs) also form functional autapses.

Training LSTM Networks With Resistive Cross-Point Devices.

In our previous work we have shown that resistive cross point devices, so called resistive processing unit (RPU) devices, can provide significant power and speed benefits when training deep fully connected networks as well as convolutional neural networks. In this work, we further extend the RPU concept for training recurrent neural networks (RNNs) namely LSTMs. We show that the mapping of recurrent layers is very similar to the mapping of fully connected layers and therefore the RPU concept can potentially provide large acceleration factors for RNNs as well.

Distinct Inhibitory Circuits Orchestrate Cortical beta and gamma Band Oscillations.

Distinct subtypes of inhibitory interneuron are known to shape diverse rhythmic activities in the cortex, but how they interact to orchestrate specific band activity remains largely unknown. By recording optogenetically tagged interneurons of specific subtypes in the primary visual cortex of behaving mice, we show that spiking of somatostatin (SOM)- and parvalbumin (PV)-expressing interneurons preferentially correlates with cortical beta and gamma band oscillations, respectively. Suppression of SOM cell spiking reduces the spontaneous low-frequency band (<30-Hz) oscillations and selectively reduces visually induced enhancement of beta oscillation.

A distinct entorhinal cortex to hippocampal CA1 direct circuit for olfactory associative learning.

Lateral and medial parts of entorhinal cortex (EC) convey nonspatial 'what' and spatial 'where' information, respectively, into hippocampal CA1, via both the indirect EC layer 2→ hippocampal dentate gyrus→CA3→CA1 and the direct EC layer 3→CA1 paths. However, it remains elusive how the direct path transfers distinct information and contributes to hippocampal learning functions. Here we report that lateral EC projection neurons selectively form direct excitatory synapses onto a subpopulation of morphologically complex, calbindin-expressing pyramidal cells (PCs) in the dorsal CA1 (dCA1), while medial EC neurons uniformly innervate all dCA1 PCs.

Selective Maturation of Temporal Dynamics of Intracortical Excitatory Transmission at the Critical Period Onset.

Although the developmental maturation of cortical inhibitory synapses is known to be a critical factor in gating the onset of critical period (CP) for experience-dependent cortical plasticity, how synaptic transmission dynamics of other cortical synapses are regulated during the transition to CP remains unknown. Here, by systematically examining various intracortical synapses within layer 4 of the mouse visual cortex, we demonstrate that synaptic temporal dynamics of intracortical excitatory synapses on principal cells (PCs) and inhibitory parvalbumin- or somatostatin-expressing cells are selectively regulated before the CP onset, whereas those of intracortical inhibitory synapses and long-range thalamocortical excitatory synapses remain unchanged. This selective maturation of synaptic dynamics results from a ubiquitous reduction of presynaptic release and is dependent on visual experience.

A Phenomenological Synapse Model for Asynchronous Neurotransmitter Release.

Neurons communicate with each other via synapses. Action potentials cause release of neurotransmitters at the axon terminal. Typically, this neurotransmitter release is tightly time-locked to the arrival of an action potential and is thus called synchronous release.

Integration or separation in the processing of facial properties--a computational view.

A face recognition system ought to read out information about the identity, facial expression and invariant properties of faces, such as sex and race. A current debate is whether separate neural units in the brain deal with these face properties individually or whether a single neural unit processes in parallel all aspects of faces. While the focus of studies has been directed toward the processing of identity and facial expression, little research exists on the processing of invariant aspects of faces.

Decentralized Multisensory Information Integration in Neural Systems.

Unlabelled: How multiple sensory cues are integrated in neural circuitry remains a challenge. The common hypothesis is that information integration might be accomplished in a dedicated multisensory integration area receiving feedforward inputs from the modalities. However, recent experimental evidence suggests that it is not a single multisensory brain area, but rather many multisensory brain areas that are simultaneously involved in the integration of information.

Experience-dependent emergence of beta and gamma band oscillations in the primary visual cortex during the critical period.

Neural oscillatory activities have been shown to play important roles in neural information processing and the shaping of circuit connections during development. However, it remains unknown whether and how specific neural oscillations emerge during a postnatal critical period (CP), in which neuronal connections are most substantially modified by neural activity and experience. By recording local field potentials (LFPs) and single unit activity in developing primary visual cortex (V1) of head-fixed awake mice, we here demonstrate an emergence of characteristic oscillatory activities during the CP.

Selective Modulation of Axonal Sodium Channel Subtypes by 5-HT1A Receptor in Cortical Pyramidal Neuron.

Serotonergic innervation of the prefrontal cortex (PFC) modulates neuronal activity and PFC functions. However, the cellular mechanism for serotonergic modulation of neuronal excitability remains unclear. We performed patch-clamp recording at the axon of layer-5 pyramidal neurons in rodent PFC slices.

Circuit motifs for contrast-adaptive differentiation in early sensory systems: the role of presynaptic inhibition and short-term plasticity.

In natural signals, such as the luminance value across of a visual scene, abrupt changes in intensity value are often more relevant to an organism than intensity values at other positions and times. Thus to reduce redundancy, sensory systems are specialized to detect the times and amplitudes of informative abrupt changes in the input stream rather than coding the intensity values at all times. In theory, a system that responds transiently to fast changes is called a differentiator.

Own-race and own-species advantages in face perception: a computational view.

The frequency to which an organism is exposed to a particular type of face influences recognition performance. For example, Asians are better in individuating Asian than Caucasian faces, known as the own-race advantage. Similarly, humans in general are better in individuating human than monkey faces, known as the own-species advantage.

The other-race and other-species effects in face perception - a subordinate-level analysis.

The ability of face discrimination is modulated by the frequency of exposure to a category of faces. In other words, lower discrimination performance was measured for infrequently encountered faces as opposed to frequently encountered ones. This phenomenon has been described in the literature: the own-race advantage, a benefit in processing own-race as opposed to the other-race faces, and the own-species advantage, a benefit in processing the conspecific type of faces as opposed to the heterospecific type.

Perceptual training continuously refines neuronal population codes in primary visual cortex.

Perceptual learning substantially improves visual discrimination and detection ability, which has been associated with visual cortical plasticity. However, little is known about the dynamic changes in neuronal response properties over the course of training. Using chronically implanted multielectrode arrays, we were able to capture day-by-day spatiotemporal dynamics of neurons in the primary visual cortex (V1) of monkeys trained to detect camouflaged visual contours.

Binocular input coincidence mediates critical period plasticity in the mouse primary visual cortex.

Classical studies on the development of ocular dominance (OD) organization in primary visual cortex (V1) have revealed a postnatal critical period (CP), during which visual inputs between the two eyes are most effective in shaping cortical circuits through synaptic competition. A brief closure of one eye during CP caused a pronounced shift of response preference of V1 neurons toward the open eye, a form of CP plasticity in the developing V1. However, it remains unclear what particular property of binocular inputs during CP is responsible for mediating this experience-dependent OD plasticity.

Design principles of the sparse coding network and the role of "sister cells" in the olfactory system of Drosophila.

Sensory systems face the challenge to represent sensory inputs in a way to allow easy readout of sensory information by higher brain areas. In the olfactory system of the fly drosopohila melanogaster, projection neurons (PNs) of the antennal lobe (AL) convert a dense activation of glomeruli into a sparse, high-dimensional firing pattern of Kenyon cells (KCs) in the mushroom body (MB). Here we investigate the design principles of the olfactory system of drosophila in regard to the capabilities to discriminate odor quality from the MB representation and its robustness to different types of noise.

The face inversion effect in non-human primates revisited - an investigation in chimpanzees (Pan troglodytes).

Faces presented upside-down are harder to recognize than presented right-side up, an effect known as the face inversion effect. With inversion the perceptual processing of the spatial relationship among facial parts is disrupted. Previous literature indicates a face inversion effect in chimpanzees toward familiar and conspecific faces.

Laterality effect for faces in chimpanzees (Pan troglodytes).

Face perception in humans is governed more by right-hemispheric than left-hemispheric neural correlate. Some but not all neurophysiological studies depict a right-side dominance for face responsive neurons in the brains of macaques. Hence, it is an open question whether and to what extent a right-hemisphere preference of processing faces exists across primate brains.