Tensor

Introduction: The Data Structure Behind Every Model

Open any PyTorch training script or TensorFlow/Keras pipeline and you will encounter the same primitive repeatedly:

• A loss value computed from predictions and labels
• Weight matrices updated by an optimizer
• Activation tensors flowing through layers
• Input batches fed to the model each training step

Framework documentation calls all of these tensors. The word sounds abstract, but the engineering reality is concrete: a tensor is how numerical data is stored, typed, shaped, and moved through a computation graph. Misunderstanding tensor shape is one of the most common sources of runtime errors in AI engineering—often appearing as cryptic messages about incompatible dimensions during matrix multiplication or convolution.

Mastering tensors is not optional background mathematics. It is the vocabulary of production deep learning.

Scalars, Vectors, Matrices—and Beyond

Mathematical physics uses “tensor” for objects that transform under coordinate changes—a richer definition than we need day to day in ML engineering. In PyTorch and TensorFlow, the term is operational: a tensor is the container type for numerical computation. That pragmatic definition is what matters when you debug a training job at 2 a.m.

Rank, Order, and Shape

Three terms appear constantly in documentation. Students must use them precisely.

Order (Rank) of a Tensor

The order or rank of a tensor is the number of dimensions (axes) it has. Equivalently, it is the length of the shape tuple.

• Shape (768,) → order 1 (vector)
• Shape (32, 768) → order 2 (matrix)
• Shape (32, 128, 768) → order 3
• Shape (32, 3, 224, 224) → order 4

Shape

Shape tells you how many elements exist along each axis, read left to right. For a transformer hidden state with shape (32, 128, 768):

• Axis 0: batch size 32 (32 sequences processed in parallel)
• Axis 1: sequence length 128 (128 tokens per sequence)
• Axis 2: hidden dimension 768 (768 features per token)

Total elements: 32 × 128 × 768 = 3,145,728 floating-point values. Shape is how engineers reason about memory footprint and whether two tensors can be combined in an operation.

Element, Axis, and Slice

An element is a single stored value at a coordinate (e.g., batch 5, channel 2, row 100, column 50). An axis is one dimension of the array. A slice fixes all but one or more axes—for example, images[0] selects the first image in a batch, producing a (3, 224, 224) tensor.

Image Tensors: Batch × Height × Width × Channels

Computer vision pipelines represent images as 4D tensors. A single grayscale image is 2D (height × width). A single RGB image adds a channel axis: 3 × height × width. Production training almost always processes a batch of images in parallel, yielding a 4D tensor.

Channel order within RGB is also standardized as R, G, B in most frameworks when channels are explicit. Preprocessing pipelines must match what the pretrained model expects—including normalization constants applied per channel.

From Pixels to Model Input

Raw image files are not tensors. Engineering pipelines perform deterministic transforms:

PyTorch Tensors

PyTorch is the dominant framework in research and an increasingly common production choice. Its central type is torch.Tensor.

Creating and Inspecting Tensors

import torch

# Scalar-like (0D): loss accumulator
loss = torch.tensor(2.47)

# Vector (1D): embedding dimension
vec = torch.randn(768)

# Matrix (2D): weight matrix
W = torch.randn(512, 768)

# 3D: batch of sequences (batch, seq_len, hidden)
hidden = torch.randn(32, 128, 768)

# 4D: image batch (NCHW)
images = torch.randn(32, 3, 224, 224)

print(images.shape)   # torch.Size([32, 3, 224, 224])
print(images.dtype)   # torch.float32
print(images.device)  # cpu (until moved)

dtype and Numerical Stability

Common dtypes in AI engineering:

• float32 — default for most training; balance of speed and precision
• float16 / bfloat16 — mixed-precision training on modern GPUs; faster, less memory
• int64 — token IDs, class labels, indexing
• bool — attention masks, padding masks

dtype mismatches (e.g., float64 weights with float32 inputs) trigger errors or implicit casts. Explicit dtype control is part of reproducible engineering.

Device Placement

GPU training requires every tensor in an operation to live on the same device. The pattern is universal:

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device)
batch = batch.to(device)

Forgetting to move inputs while the model sits on GPU produces the infamous error: Expected all tensors to be on the same device.

Autograd: Tensors That Remember

Training tensors often set requires_grad=True so PyTorch can compute gradients via automatic differentiation. Loss scalars drive backpropagation; weight matrices accumulate .grad attributes. Inference-only tensors typically disable gradients for speed and memory savings:

with torch.no_grad():
    predictions = model(images)

TensorFlow Tensors

TensorFlow (with Keras as the high-level API) remains widely deployed in enterprise and mobile/edge stacks. Its core type is tf.Tensor, often created implicitly through Keras layers and tf.data pipelines.

Creating and Inspecting Tensors

import tensorflow as tf

# Explicit tensor creation
scalar = tf.constant(2.47)
vec = tf.random.normal([768])
W = tf.random.normal([512, 768])
hidden = tf.random.normal([32, 128, 768])
images = tf.random.normal([32, 224, 224, 3])  # NHWC

print(images.shape)   # (32, 224, 224, 3)
print(images.dtype)   # <dtype: 'float32'>

Eager Execution vs Graph Mode

Modern TensorFlow runs in eager mode by default—operations execute immediately, similar to PyTorch’s imperative style. @tf.function traces computations into graphs for performance. Engineers debugging shape issues start in eager mode; they optimize with graphs once shapes are stable.

Keras Integration

Keras models accept NumPy arrays or tensors and return tensors. Layer definitions encode shape transformations:

model = tf.keras.Sequential([
    tf.keras.layers.Conv2D(32, 3, activation="relu", input_shape=(224, 224, 3)),
    tf.keras.layers.GlobalAveragePooling2D(),
    tf.keras.layers.Dense(10, activation="softmax"),
])

Note input_shape=(224, 224, 3)—NHWC without the batch dimension. Keras inserts None for batch size automatically.

Batching: Why the First Dimension Matters

Neural networks process batches of examples in parallel to saturate GPU throughput. Conventionally, axis 0 is the batch dimension unless documentation states otherwise.

• Training: batch size 32, 64, or 128 is typical; limited by GPU memory
• Inference: batch size 1 for latency-sensitive APIs; larger batches for offline scoring
• Variable-length sequences: padding and attention masks align batch items to a common length

When you squeeze or unsqueeze dimensions, you are changing tensor order. Removing the batch axis accidentally—images.squeeze() when batch size is 1—turns a 4D batch into a 3D tensor and breaks the next layer expecting 4D input.

Reshaping, Broadcasting, and Views

Engineers constantly change tensor geometry without changing underlying data semantics.

Reshape and View

reshape / view (PyTorch) and reshape (TensorFlow) reinterpret the same elements with a new shape. Total element count must match. Flattening an image batch for a fully connected layer:

# PyTorch: (32, 3, 224, 224) -> (32, 150528)
flat = images.view(32, -1)

Broadcasting

Broadcasting allows operations between tensors of different but compatible shapes by virtually expanding dimensions. Adding a per-channel bias to an image batch relies on broadcasting. Incompatible shapes produce errors—another reason shape literacy matters before Matrix Multiplication.

Permute and Transpose

Switching between NCHW and NHWC is a permutation, not a copy of pixel values in a different order without care:

# PyTorch: NCHW -> NHWC
images_nhwc = images.permute(0, 2, 3, 1)

Tensor Operations Every Engineer Uses

Debugging Tensors in Production Workflows

When a model fails or metrics look wrong, tensor inspection is the first diagnostic step:

1. Print shape at model entry and exit—confirm expectations match reality
2. Check dtype—uint8 images not normalized will destroy training
3. Verify device—CPU/GPU mismatch errors are immediate; silent CPU fallback can hide performance issues
4. Inspect value ranges—tensor.min(), tensor.max(), torch.isnan(tensor).any()
5. Confirm channel order—visualize a single image after preprocessing
6. Track memory—4D float32 batches consume VRAM quickly; order-5 video tensors more so

Common Misconceptions

Quick Knowledge Check

Further Reading & References

Official Documentation

• PyTorch — Tensors and Autograd (pytorch.org/docs)
• TensorFlow — Introduction to Tensors (tensorflow.org/guide/tensor)
• torchvision transforms — Image preprocessing and NCHW conventions
• tf.keras.layers.Conv2D — Input shape and channels-last behavior

Books & Tutorials

• Deep Learning with PyTorch — Eli Stevens, Luca Antiga, and Thomas Viehmann. Practical tensor-centric workflows.
• Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow — Aurélien Géron. TensorFlow tensor and shape fundamentals.
• PyTorch 60-Minute Blitz — Official quickstart for tensor operations and neural network basics.

Related Lectures in This Module

• Scalars — Zero-dimensional tensors and numerical primitives
• Vectors — One-dimensional tensors and embeddings
• Matrices — Two-dimensional tensors and linear maps
• Matrix Multiplication — The operation that connects tensor shapes to layer design