Track 1. General topics of data analysis

Tensor Networks and Deep Learning — Bridge Between Tensor Networks and Deep Neural Networks: From Fundamentals to Real Applications.

July 28, 2017 9:15 am - 10:15 am

Bookmark and Share

Andrzej Cichocki

Tensor decompositions (TD) and their generalizations tensor networks (TN) are promising, and emerging
tools in Machine Learning (ML), especially in Deep Learning (DL), since  input/output  data  outputs
in hidden layers can be naturally represented and described  as higher-order tensors and most operations
can be performed using  optimized linear/multilinear algebra.

I will present a brief overview of  tensor decomposition and tensor networks architectures and associated
learning algorithms.  I will also  discuss several applications  of tensor networks in  Signal Processing,
Machine Learning, both in supervised and unsupervised learning and possibility of dramatic  reduction
of set of parameters in  state-of-the arts deep CNN, typically, from hundreds millions to  tens  of
thousands of parameters. We focus on  novel  (Quantized) Tensor Train-Tucker (QTT-Tucker)
and Quantized Hierarchical Tucker (QHT) tensor network  models  for higher order tensors
(tensors of order  at least four or higher). Moreover, we present   tensor sketching for efficient
dimensionality reduction which avoid curse of dimensionality.

Tensor Train-Tucker and HT models will be naturally extended to MERA
(Multiscale Entanglement Renormalization Ansatz) models, TTNS (Tree Tensor Network States)
and  PEPS/PEPO  and  other 2D/3D tensor networks,  with improved expressive power of
deep learning in convolutional neural networks (DCNN) and inspiration to generate novel architectures
of deep and semi-shallow neural networks. Furthermore, we will be show how to apply tensor networks
to higher order multiway, partially restricted Boltzmann Machine (RBM) with substantial reduction
of set of learning parameters.


Cichocki, A., Lee, N., Oseledets, I., Phan, A. H., Zhao, Q., & Mandic, D. P. (2016).
Tensor networks for dimensionality reduction and large-scale optimization:
Part 1 low-rank tensor decompositions.
Foundations and Trends® in Machine Learning9(4-5), 249-429.

Cichocki, A., Phan, A. H., Zhao, Q., Lee, N., Oseledets, I., Sugiyama, M., & Mandic, D. P. (2017).
Tensor Networks for Dimensionality Reduction and Large-scale Optimization:
Part 2 Applications and Future Perspectives. 
Foundations and Trends® in Machine Learning
9(6), 431-673.

Cichocki, A., Mandic, D., De Lathauwer, L., Zhou, G., Zhao, Q., Caiafa, C., & Phan, H. A. (2015).
“Tensor decompositions for signal processing applications:
From two-way to multiway component analysis”.
IEEE Signal Processing Magazine32(2), 145-163.