Before a neural network can learn anything, its data has to live somewhere. That “somewhere” is a tensor — a container for numbers. Every image, sentence, and spreadsheet you feed a model first gets turned into one, and every layer inside the model is just a small set of operations (add, multiply, reshape) applied to tensors. Get comfortable with tensors now, and the rest of deep learning is just chaining them together.

What’s a tensor?

At its core, a tensor is a container for numerical data. You already know one special case: a matrix, which is a tensor with two axes. A tensor is simply the generalization of a matrix to any number of axes — in tensor-speak, a dimension is usually called an axis, and the number of axes is called the tensor’s rank.

  flowchart LR
  A["Scalar (rank-0)
a single number"] --> B["Vector (rank-1)
a list of numbers"]
  B --> C["Matrix (rank-2)
rows and columns"]
  C --> D["Rank-3 tensor
a stack of matrices"]

Scalars (rank-0)

A tensor with just one number, and 0 axes:

import numpy as np
 
x = np.array(12)
print(x.ndim)   # 0

import numpy as np
 
x = np.array(12)
print(x.ndim)   # 0

Vectors (rank-1)

An array of numbers, with exactly one axis. This vector has 5 entries, so it’s called a “5-dimensional vector” — don’t confuse that with a “5D tensor” (which would have five axes, not five entries):

x = np.array([12, 3, 6, 14, 7])
print(x.ndim)   # 1

x = np.array([12, 3, 6, 14, 7])
print(x.ndim)   # 1

Matrices (rank-2)

An array of vectors — two axes, conventionally called rows and columns:

x = np.array([[5, 78, 2, 34, 0],
              [6, 79, 3, 35, 1],
              [7, 80, 4, 36, 2]])
print(x.ndim)   # 2
print(x.shape)  # (3, 5)

x = np.array([[5, 78, 2, 34, 0],
              [6, 79, 3, 35, 1],
              [7, 80, 4, 36, 2]])
print(x.ndim)   # 2
print(x.shape)  # (3, 5)

Rank-3 and higher

Pack several matrices of the same shape into a new array, and you get a rank-3 tensor — visually, a cube of numbers. Pack rank-3 tensors into an array, and you get rank-4, and so on. In practice, deep learning mostly uses tensors of rank 0 to 4 (rank 5 shows up for video).

The three attributes of every tensor

1. Rank (ndimndim) — how many axes the tensor has.
2. Shape — a tuple of integers, one per axis, describing how many entries live along it. A vector has a shape like (5,)(5,); a scalar has an empty shape ()().
3. Dtype — what kind of numbers it holds (float32float32, uint8uint8, int64int64, …).

from tensorflow.keras.datasets import mnist
 
(train_images, train_labels), _ = mnist.load_data()
 
print(train_images.ndim)   # 3  -- an array of 2D images
print(train_images.shape)  # (60000, 28, 28)
print(train_images.dtype)  # uint8

from tensorflow.keras.datasets import mnist
 
(train_images, train_labels), _ = mnist.load_data()
 
print(train_images.ndim)   # 3  -- an array of 2D images
print(train_images.shape)  # (60000, 28, 28)
print(train_images.dtype)  # uint8

train_imagestrain_images is a rank-3 tensor: 60,000 grayscale images, each a 28×28 grid of pixel values from 0 to 255. The first axis of a data tensor is (almost) always the samples axis — one entry per example in your dataset.

Real-world tensor shapes

You’ll see the same handful of shapes over and over:

• Vector data — rank-2, (samples, features)(samples, features). One row per example.
• Timeseries / sequences — rank-3, (samples, timesteps, features)(samples, timesteps, features).
• Images — rank-4, (samples, height, width, channels)(samples, height, width, channels).
• Video — rank-5, (samples, frames, height, width, channels)(samples, frames, height, width, channels).

Element-wise operations

relurelu and ++ are element-wise: each entry in the output only depends on the matching entry (or entries) in the input. That’s why NumPy can run them so fast — it hands the loop off to a highly optimized, vectorized implementation (BLAS) instead of looping in Python:

import numpy as np
import time
 
x = np.random.random((20, 100))
y = np.random.random((20, 100))
 
t0 = time.time()
z = x + y
z = np.maximum(z, 0.)
print(f"Vectorized: {time.time() - t0:.5f}s")
 
def naive_add_relu(x, y):
    x = x.copy()
    for i in range(x.shape[0]):
        for j in range(x.shape[1]):
            x[i, j] = max(x[i, j] + y[i, j], 0)
    return x
 
t0 = time.time()
naive_add_relu(x, y)
print(f"Naive Python loop: {time.time() - t0:.5f}s")

import numpy as np
import time
 
x = np.random.random((20, 100))
y = np.random.random((20, 100))
 
t0 = time.time()
z = x + y
z = np.maximum(z, 0.)
print(f"Vectorized: {time.time() - t0:.5f}s")
 
def naive_add_relu(x, y):
    x = x.copy()
    for i in range(x.shape[0]):
        for j in range(x.shape[1]):
            x[i, j] = max(x[i, j] + y[i, j], 0)
    return x
 
t0 = time.time()
naive_add_relu(x, y)
print(f"Naive Python loop: {time.time() - t0:.5f}s")

The vectorized version is routinely 100× faster or more — one reason deep learning frameworks lean so heavily on tensor operations instead of raw Python loops.

Broadcasting

A DenseDense layer computes dot(input, W) + bdot(input, W) + b — but bb is a vector and the result of the dot product is a matrix. How does ++ even work when the shapes don’t match?

When there’s no ambiguity, NumPy broadcasts the smaller tensor to match the larger one:

1. Axes are added to the smaller tensor until its rank matches the larger one.
2. The smaller tensor is then virtually repeated along those new axes.

import numpy as np
 
X = np.array([[1., 2., 3.],
              [4., 5., 6.]])       # shape (2, 3)
y = np.array([10., 20., 30.])      # shape (3,)
 
z = X + y
print(z)
# [[11. 22. 33.]
#  [14. 25. 36.]]

import numpy as np
 
X = np.array([[1., 2., 3.],
              [4., 5., 6.]])       # shape (2, 3)
y = np.array([10., 20., 30.])      # shape (3,)
 
z = X + y
print(z)
# [[11. 22. 33.]
#  [14. 25. 36.]]

yy is treated as if it were repeated once per row of XX — but NumPy never actually copies the memory; the repetition is purely algorithmic. This is exactly how a bias vector bb gets added to every sample in a batch inside a DenseDense layer.

Reshaping and transposition

Reshaping rearranges a tensor’s entries into a new shape, keeping the total number of values the same. It’s what turns a (60000, 28, 28)(60000, 28, 28) image tensor into (60000, 784)(60000, 784) before feeding it to a DenseDense layer:

import numpy as np
 
x = np.array([[0., 1.],
              [2., 3.],
              [4., 5.]])
print(x.shape)              # (3, 2)
 
x = x.reshape((6, 1))
print(x.ravel())            # [0. 1. 2. 3. 4. 5.]
 
x = x.reshape((2, 3))
print(x)
# [[0. 1. 2.]
#  [3. 4. 5.]]

import numpy as np
 
x = np.array([[0., 1.],
              [2., 3.],
              [4., 5.]])
print(x.shape)              # (3, 2)
 
x = x.reshape((6, 1))
print(x.ravel())            # [0. 1. 2. 3. 4. 5.]
 
x = x.reshape((2, 3))
print(x)
# [[0. 1. 2.]
#  [3. 4. 5.]]

Transposition is a special case of reshaping: it swaps rows and columns, so x[i, :]x[i, :] becomes x[:, i]x[:, i]:

import numpy as np
 
x = np.zeros((300, 20))
x = np.transpose(x)
print(x.shape)   # (20, 300)

import numpy as np
 
x = np.zeros((300, 20))
x = np.transpose(x)
print(x.shape)   # (20, 300)

The tensor dot product

The dot product (np.dotnp.dot) combines two tensors into one, and it’s the operation behind every DenseDense layer’s W · xW · x. Shapes must be compatible: for two matrices, dot(x, y)dot(x, y) only works if x.shape[1] == y.shape[0]x.shape[1] == y.shape[0], and the result has shape (x.shape[0], y.shape[1])(x.shape[0], y.shape[1]):

import numpy as np
 
a = np.random.random((2, 3))
b = np.random.random((3, 4))
 
z = np.dot(a, b)
print(z.shape)   # (2, 4)  -- inner dimension (3) disappears

import numpy as np
 
a = np.random.random((2, 3))
b = np.random.random((3, 4))
 
z = np.dot(a, b)
print(z.shape)   # (2, 4)  -- inner dimension (3) disappears

The same rule generalizes to higher ranks: (a, b, c, d) · (d,) -> (a, b, c)(a, b, c, d) · (d,) -> (a, b, c), and (a, b, c, d) · (d, e) -> (a, b, c, e)(a, b, c, d) · (d, e) -> (a, b, c, e) — the shared inner dimension always cancels out.

Visualize it

Reshaping rearranges the same numbers into a new grid; broadcasting virtually repeats a smaller tensor so an element-wise operation can apply it to every row of a larger one. Watch both happen below:

Mini-checkpoint

A batch of 128 color images, each 64×64 pixels, is stored as one tensor. What’s its rank, and what’s its shape (channels-last)?

• Rank 4, shape (128, 64, 64, 3)(128, 64, 64, 3) — samples, height, width, color channels.

Next

Now that you can describe any data as a tensor, the next question is: how does a network actually learn the right numbers to put inside its weight tensors? That’s Introduction to Neural Networks (The Perceptron) — the simplest possible learner, built from exactly the tensor operations you just saw.

🧪 Try It Yourself

Exercise 1 – Rank, Shape, and Dtype

Exercise 2 – Broadcasting a Bias Vector

Exercise 3 – Dot Product Shape Rule