Tensor decompositions (TD) and their generalizations tensor networks (TN) are promising, and emerging
tools in Machine Learning (ML), especially in Deep Learning (DL), since input/output data outputs
in hidden layers can be naturally represented and described as higher-order tensors and most operations
can be performed using optimized linear/multilinear algebra.
I will present a brief overview of tensor decomposition and tensor networks architectures and associated
learning algorithms. I will also discuss several applications of tensor networks in Signal Processing,
Machine Learning, both in supervised and unsupervised learning and possibility of dramatic reduction
of set of parameters in state-of-the arts deep CNN, typically, from hundreds millions to tens of
thousands of parameters. We focus on novel (Quantized) Tensor Train-Tucker (QTT-Tucker)
and Quantized Hierarchical Tucker (QHT) tensor network models for higher order tensors
(tensors of order at least four or higher). Moreover, we present tensor sketching for efficient
dimensionality reduction which avoid curse of dimensionality.
Tensor Train-Tucker and HT models will be naturally extended to MERA
(Multiscale Entanglement Renormalization Ansatz) models, TTNS (Tree Tensor Network States)
and PEPS/PEPO and other 2D/3D tensor networks, with improved expressive power of
deep learning in convolutional neural networks (DCNN) and inspiration to generate novel architectures
of deep and semi-shallow neural networks. Furthermore, we will be show how to apply tensor networks
to higher order multiway, partially restricted Boltzmann Machine (RBM) with substantial reduction
of set of learning parameters.
References:
Cichocki, A., Lee, N., Oseledets, I., Phan, A. H., Zhao, Q., & Mandic, D. P. (2016).
Tensor networks for dimensionality reduction and large-scale optimization:
Part 1 low-rank tensor decompositions.
Foundations and Trends® in Machine Learning, 9(4-5), 249-429.
Cichocki, A., Phan, A. H., Zhao, Q., Lee, N., Oseledets, I., Sugiyama, M., & Mandic, D. P. (2017).
Tensor Networks for Dimensionality Reduction and Large-scale Optimization:
Part 2 Applications and Future Perspectives.
Foundations and Trends® in Machine Learning, 9(6), 431-673.
Cichocki, A., Mandic, D., De Lathauwer, L., Zhou, G., Zhao, Q., Caiafa, C., & Phan, H. A. (2015).
“Tensor decompositions for signal processing applications:
From two-way to multiway component analysis”.
IEEE Signal Processing Magazine, 32(2), 145-163.