Tensor decompositions (TD) and their generalizations tensor networks (TN) are promising, and emerging

tools in Machine Learning (ML), especially in Deep Learning (DL), since input/output data outputs

in hidden layers can be naturally represented and described as higher-order tensors and most operations

can be performed using optimized linear/multilinear algebra.

I will present a brief overview of tensor decomposition and tensor networks architectures and associated

learning algorithms. I will also discuss several applications of tensor networks in Signal Processing,

Machine Learning, both in supervised and unsupervised learning and possibility of dramatic reduction

of set of parameters in state-of-the arts deep CNN, typically, from hundreds millions to tens of

thousands of parameters. We focus on novel (Quantized) Tensor Train-Tucker (QTT-Tucker)

and Quantized Hierarchical Tucker (QHT) tensor network models for higher order tensors

(tensors of order at least four or higher). Moreover, we present tensor sketching for efficient

dimensionality reduction which avoid curse of dimensionality.

Tensor Train-Tucker and HT models will be naturally extended to MERA

(Multiscale Entanglement Renormalization Ansatz) models, TTNS (Tree Tensor Network States)

and PEPS/PEPO and other 2D/3D tensor networks, with improved expressive power of

deep learning in convolutional neural networks (DCNN) and inspiration to generate novel architectures

of deep and semi-shallow neural networks. Furthermore, we will be show how to apply tensor networks

to higher order multiway, partially restricted Boltzmann Machine (RBM) with substantial reduction

of set of learning parameters.

References:

Cichocki, A., Lee, N., Oseledets, I., Phan, A. H., Zhao, Q., & Mandic, D. P. (2016).

Tensor networks for dimensionality reduction and large-scale optimization:

Part 1 low-rank tensor decompositions.

*Foundations and Trends® in Machine Learning*, *9*(4-5), 249-429.

Cichocki, A., Phan, A. H., Zhao, Q., Lee, N., Oseledets, I., Sugiyama, M., & Mandic, D. P. (2017).

Tensor Networks for Dimensionality Reduction and Large-scale Optimization:

Part 2 Applications and Future Perspectives. *
Foundations and Trends® in Machine Learning*,

*9*(6), 431-673.

Cichocki, A., Mandic, D., De Lathauwer, L., Zhou, G., Zhao, Q., Caiafa, C., & Phan, H. A. (2015).

“Tensor decompositions for signal processing applications:

From two-way to multiway component analysis”.

*IEEE Signal Processing Magazine*, *32*(2), 145-163.