Fractal Activation Functions Achieve 2.6x Faster Convergence in Echo State Networks

Reservoir computing typically relies on smooth activation functions, a limitation that hinders performance in challenging real-world applications such as disaster response and pharmaceutical modelling. Rae Chipera from the National University School of Technology and Engineering, along with Jenny Du and Irene Tsapara, systematically investigates the potential of non-smooth activation functions, including chaotic and fractal variants, within echo state networks. Their comprehensive analysis of over 36,000 network configurations demonstrates that these functions not only maintain essential stability properties but frequently surpass the performance of traditional smooth activations in both convergence speed and tolerance to varying network parameters. Notably, the team finds that the Cantor function exhibits remarkably stable behaviour at spectral radii ten times greater than those typically tolerated by standard functions, while also converging 2. 6times faster than commonly used activations like tanh and ReLU.

This research establishes a theoretical basis for understanding these discrete functions and reveals that the topological properties of activation functions, rather than simple continuity, are key determinants of network stability, opening new avenues for designing more robust and efficient reservoir computing systems. Reservoir Size and Activation Function Impact This research presents a detailed analysis of the Echo State Property (ESP) in reservoir computing, focusing on how different activation functions and network size affect performance. The study reveals that network size significantly impacts the ESP, with many activation functions losing stability as the network grows. However, continuous, compressive preprocessing, such as those using tanh, logistic sigmoid, and continuous Mandelbrot variants, consistently maintain the ESP across a wide range of network sizes. Conversely, discrete activation functions, like the discrete Mandelbrot and Cantor set, suffer from quantization effects that limit performance in larger networks. The crowding ratio, a measure of discretization, appears to be a key indicator of performance. Functions with discontinuities consistently fail as network size increases, and Brownian motion introduces excessive noise for reliable computation. The Cantor function exhibits unique convergence behavior, suggesting potential viability under specific conditions. Importantly, the experimental results strongly support theoretical predictions about the importance of preprocessing topology and the limitations imposed by quantization. This analysis provides strong guidance for selecting activation functions for reservoir computing, recommending continuous, compressive functions, especially for large-scale networks. The results highlight the importance of carefully considering network size and the potential limitations of quantization effects. The research contributes to a deeper understanding of the ESP, informing the development of more robust and efficient reservoir computing algorithms, and has implications for hardware implementations that might benefit from using analog or mixed-signal circuits to implement continuous activation functions. Non-Smooth Activations Enhance Echo State Networks This research systematically investigates the potential of non-smooth activation functions within echo state networks, challenging the conventional reliance on smooth, globally Lipschitz continuous functions. Through extensive testing across a vast parameter space, the team demonstrates that several non-smooth functions not only maintain the Echo State Property, a crucial characteristic for reliable computation, but frequently surpass the performance of traditional smooth activations in terms of convergence speed and tolerance to high spectral radii. Notably, the Cantor function exhibits exceptional stability, maintaining the Echo State Property at spectral radii an order of magnitude beyond typical limits for smooth functions and converging 2. 6times faster than tanh and ReLU. The study introduces a theoretical framework, defining a Degenerate Echo State Property to account for the stability of discrete-output functions and proving its relationship to the traditional Echo State Property. Analysis reveals a critical crowding ratio, the relationship between reservoir size and quantization levels, that predicts failure thresholds for discrete activations, and highlights the importance of preprocessing topology in maintaining stability. Monotone, compressive preprocessing consistently preserves the Echo State Property, while dispersive or discontinuous preprocessing leads to failures. While the research establishes the viability of non-smooth functions, the underlying mechanisms responsible for the superior performance of certain fractal functions remain unclear, indicating a need for further investigation into the interplay between geometric properties and reservoir dynamics. The authors acknowledge that the performance of binary functions, such as the Cantor set, degrades at larger scales, representing a limitation for certain applications.

Cantor Function Boosts Reservoir Computing Performance Researchers have achieved a significant breakthrough in reservoir computing by demonstrating that non-smooth activation functions can not only maintain the Echo State Property (ESP), crucial for stable network behavior, but also outperform traditional smooth functions in certain scenarios. The work involved a comprehensive investigation across 36,610 reservoir configurations, systematically testing chaotic, stochastic, and fractal activation functions.

Results demonstrate that the Cantor function, a continuous function that is almost everywhere flat, maintains ESP-consistent behavior up to a spectral radius of approximately 10, an order of magnitude beyond the typical stable range for smooth activations. Notably, the Cantor function achieved convergence 2. 6times faster than both tanh and ReLU functions, indicating a substantial improvement in computational efficiency. Further testing revealed that the Cantor function maintains partial convergence even at a spectral radius of 100. These findings challenge the conventional reliance on smooth activation functions and suggest that alternative dynamics can be advantageous for reservoir computing.

The team identified a critical crowding ratio, which predicts failure thresholds for discrete activations, providing insight into the stability limits of these systems. The research establishes a theoretical framework for quantized activation functions, defining a Degenerate Echo State Property (d-ESP) that captures stability for discrete-output functions and proving that d-ESP implies traditional ESP. Analysis reveals that preprocessing topology, specifically whether it is monotone and compressive, is a key determinant of stability, while dispersive or discontinuous preprocessing triggers failures. These findings suggest that boundedness, rather than strict continuity, may be sufficient for maintaining ESP, opening new avenues for designing robust and efficient reservoir computing systems.

Reservoir Parameter Sweeps and Echo State Property Evaluation This work pioneers a systematic investigation of non-smooth activation functions within echo state networks, challenging the conventional reliance on smooth, Lipschitz continuous functions. Researchers conducted comprehensive parameter sweeps across 36,610 reservoir configurations, meticulously testing chaotic, stochastic, and fractal variants to determine their impact on the Echo State Property (ESP). The study employed sparse random matrices, scaling them to achieve precise target spectral radii, and confirmed these radii through eigenvalue decomposition. To assess ESP compliance, the team evaluated configurations for up to 2,000 timesteps, distinguishing slow convergence from genuine ESP violation. The methodology involved rigorous convergence testing from two initial conditions, a zero state and random initialization, using Gaussian, uniform, and sparse input distributions. This approach, mirroring previous bifurcation analysis, justified focusing on realistic input classes while ensuring statistical robustness with 1000 independent trials per configuration across seven reservoir sizes ranging from 1 to 2000 neurons. Researchers measured convergence rate by calculating the time required for the difference between reservoir states to fall below a threshold, enabling direct comparison of transient dynamics across different activation functions. To establish operational boundaries, the team systematically varied spectral radius and leak rate, revealing that the Cantor function maintains ESP-consistent behavior up to spectral radii an order of magnitude beyond typical bounds for smooth functions, while the logistic sigmoid remained stable throughout the grid. Detailed analysis revealed critical transitions in stability as parameters varied, demonstrating that preprocessing topology, rather than continuity, determines stability. The study identified a critical interplay between leak rate and activation function type, with discontinuous functions requiring smaller leak rates for stability and compressive maps exhibiting insensitivity within a specific range. 👉 More information 🗞 Beyond Lipschitz Continuity and Monotonicity: Fractal and Chaotic Activation Functions in Echo State Networks 🧠 ArXiv: https://arxiv.org/abs/2512.14675 Tags: