Publication

Stable tensor neural networks for efficient deep learning

Downloadable Content

Persistent URL
Last modified
  • 06/25/2025
Type of Material
Authors
    Elizabeth Newman, Emory UniversityLior Horesh, IBM TJ Watson Research CenterHaim Avron, Tel Aviv UniversityMisha E. Kilmer, Tufts University
Language
  • English
Date
  • 2024
Publisher
  • Frontiers
Publication Version
Copyright Statement
  • © 2024 Newman, Horesh, Avron and Kilmer.
License
Final Published Version (URL)
Title of Journal or Parent Work
Volume
  • 7
Start Page
  • 1363978
Grant/Funding Information
  • The author(s) declare financial support was received for the research, authorship, and/or publication of this article. This work was partially funded by the Exploratory Science program at IBM. EN's work was partially supported by the National Science Foundation (NSF) under grants [DMS-2309751]. HA's work was partially funded an IBM Faculty Award. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Abstract
  • Learning from complex, multidimensional data has become central to computational mathematics, and among the most successful high-dimensional function approximators are deep neural networks (DNNs). Training DNNs is posed as an optimization problem to learn network weights or parameters that well-approximate a mapping from input to target data. Multiway data or tensors arise naturally in myriad ways in deep learning, in particular as input data and as high-dimensional weights and features extracted by the network, with the latter often being a bottleneck in terms of speed and memory. In this work, we leverage tensor representations and processing to efficiently parameterize DNNs when learning from high-dimensional data. We propose tensor neural networks (t-NNs), a natural extension of traditional fully-connected networks, that can be trained efficiently in a reduced, yet more powerful parameter space. Our t-NNs are built upon matrix-mimetic tensor-tensor products, which retain algebraic properties of matrix multiplication while capturing high-dimensional correlations. Mimeticity enables t-NNs to inherit desirable properties of modern DNN architectures. We exemplify this by extending recent work on stable neural networks, which interpret DNNs as discretizations of differential equations, to our multidimensional framework. We provide empirical evidence of the parametric advantages of t-NNs on dimensionality reduction using autoencoders and classification using fully-connected and stable variants on benchmark imaging datasets MNIST and CIFAR-10.
Author Notes
Keywords
Research Categories
  • Mathematics

Tools

Relations

In Collection:

Items

Thumbnail Title File Description Date Uploaded Visibility Actions
file details Publication File - w9cm4.pdf Primary Content 2025-06-04 Public Download