Dimentionality#
Dimentionality is a crucial property of the tensor, so it’s important to know the torch tools to manage dimentionality.
import torch
from random import randint
Get#
There are two methods to get torch.Tensor dimentionality to use torch.Tensor.shape attribute or to use torch.Tensor.size methods.
In the follwing cells both methods were applied to the test_tensor.
test_tensor = torch.zeros([randint(2, 10) for i in range(randint(3, 6))])
test_tensor.shape
torch.Size([8, 10, 9, 4, 3])
test_tensor.size()
torch.Size([8, 10, 9, 4, 3])
torch.Size#
torch.Size object keeps dimentionality in it. It have some features:
Typical Python indexing methods.
tens = torch.zeros([randint(2,6) for i in range(7)])
print("Original shape", tens.shape)
print("Shape slice", tens.shape[1:5])
Original shape torch.Size([6, 6, 5, 4, 2, 3, 2])
Shape slice torch.Size([6, 5, 4, 2])
Some operators will have meaning when used with certain types of data.
For example, == works with tuple or list.
tens = torch.Tensor(4,3)
(
tens.shape == (4, 3),
tens.shape == [3, 1]
)
(True, False)
Null dimentional tensors#
In PyTorch, there are “null-dimensional” tensors, which are tensors with a single value. In such cases, the tensor appears without any square brackets denoting its dimensionality. For example, when we print the tensor, it doesn’t show any brackets to indicate it’s a single value. When we print its shape, it displays an empty list, signifying that the tensor has no dimensions.
example_tensor = torch.tensor(3)
print(example_tensor)
print(example_tensor.shape)
tensor(3)
torch.Size([])
Reshape like#
The transformations discussed in this section allow you to change the dimensionality of a tensor while preserving the order of its elements. The order is determined by filling elements from the innermost dimension first, then moving to the next dimension, and so on. This order is maintained in the reshaped tensor but adapted to the new dimensionality.
There are two operations in Torch that allow you to perform such transformations:
torch.Tensor.view: The fastest method, but it requires the tensor to be contiguous.torch.Tensor.reshape(or the identical functiontorch.reshape): If the tensor is contiguous, it will still usetorch.Tensor.view. Otherwise, it copies the tensor to make it contiguous before applyingtorch.view.
Find out more int particular page.
As an example, consider a flat tensor (vector).
example = torch.arange(12)
example
tensor([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11])
Using the torhc.tensor.reshape method, we composed a 3x4 matrix from the elements of the original tensor.
example = example.reshape([3, 4])
example
tensor([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]])
Using the torch.reshape function, we obtained a 3-dimensional tensor with the original elements.
torch.reshape(example, (2,3,2))
tensor([[[ 0, 1],
[ 2, 3],
[ 4, 5]],
[[ 6, 7],
[ 8, 9],
[10, 11]]])
Transpose/permute#
Transposing is an operation that reindexes the elements of the original tensor. For simplicity, consider the two-dimensional case: if we have a matrix \(\left[x_{ij}\right]_{n \times m}\), after transposing, we get a matrix \(\left[x_{ji}\right]_{m \times n}\). Learn more on the specific page.
For operations like that torch have tools:
torch.transposefunction,torch.Tensor.trasposemethod to change indexes for two sepcified axis.torch.Tensor.Tattribute that switches dimentions for two dimentional tensor.torch.premutefunction,torch.Tensor.permutemethod to set specific order of dimentions not only switch two selected.
As example consider matrix defined in the following cell:
example = torch.arange(12).reshape([3, 4])
example
tensor([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]])
Applying transpose(0, 1) specifies switching dimension 0 with dimension 1:
example.transpose(0, 1)
tensor([[ 0, 4, 8],
[ 1, 5, 9],
[ 2, 6, 10],
[ 3, 7, 11]])
But permute(1, 0) gives the same output. It specifies that the index at position 1 in the input should move to position 0 in the output, and the index at position 0 in the original tensor should move to position 1 in the result.
example.permute(1, 0)
tensor([[ 0, 4, 8],
[ 1, 5, 9],
[ 2, 6, 10],
[ 3, 7, 11]])
Add dimention#
Adding an extra dimension to a tensor can be beneficial, particularly when performing matrix multiplication.
All methods of creating a new dimension will result in a new axis being created at the specified position. Each element at the specified position will be transformed into a one-element array along this new axis. Find out more in the specific page.
The following example is a quick overview of this operation. For a matrix, it inserts a rows dimension using a None index and a columns dimension using the unsqueeze method.
show_tensor = torch.arange(15).reshape(5,3)
display(show_tensor)
display(show_tensor[:, None])
display(show_tensor.unsqueeze(-1))
tensor([[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8],
[ 9, 10, 11],
[12, 13, 14]])
tensor([[[ 0, 1, 2]],
[[ 3, 4, 5]],
[[ 6, 7, 8]],
[[ 9, 10, 11]],
[[12, 13, 14]]])
tensor([[[ 0],
[ 1],
[ 2]],
[[ 3],
[ 4],
[ 5]],
[[ 6],
[ 7],
[ 8]],
[[ 9],
[10],
[11]],
[[12],
[13],
[14]]])
Remove dimention#
The torch.squeeze function and the torch.Tensor.squeeze method remove dimensions from a tensor that have a size of one. These dimensions are considered “ambiguous” as they contain only a single value. Find out more in corresponding page.
Consider a tensor containing a few 3D objects. These objects are actually single rows wrapped as 3D layers, with each element being a 1-element vector.
example_tensor = torch.arange(15).reshape(3, 1, 5, 1)
example_tensor
tensor([[[[ 0],
[ 1],
[ 2],
[ 3],
[ 4]]],
[[[ 5],
[ 6],
[ 7],
[ 8],
[ 9]]],
[[[10],
[11],
[12],
[13],
[14]]]])
By applying the squeeze method we got just a matrix, so any 3D tensor with ambiguous values becomes just a vector.
example_tensor.squeeze()
tensor([[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14]])
Add padding#
There are cases when you’ll need to add elements along certain axes, regardless of their exact values, just to make the elements compatible for passing to the next layer of the network. For this purpose, you can use torch.nn.functional.pad.
Find out more in official documentaion page.
Note: There are special layers in Torch that implement padding—it’s preferable to use them.
In the first argument, you specify the tensor that needs to be transformed, and in the second, an array of numbers \((x_1^l, x_1^r, x_2^l, x_2^r, \dots, x_m^l, x_m^r)\), where \(m\) is the number of dimensions of the tensor being transformed. \(x_k^l\) is the number of elements to be added at the beginning (left) of the \(k\)-th dimension, and \(x_k^r\) is the number of elements to be added at the end (right side).
As an example, consider a matrix of ones.
example = torch.ones(5, 5)
example
tensor([[1., 1., 1., 1., 1.],
[1., 1., 1., 1., 1.],
[1., 1., 1., 1., 1.],
[1., 1., 1., 1., 1.],
[1., 1., 1., 1., 1.]])
The following cell adds one element at the beginning of each row and two elements at the end of each row:
torch.nn.functional.pad(example, (1, 2))
tensor([[0., 1., 1., 1., 1., 1., 0., 0.],
[0., 1., 1., 1., 1., 1., 0., 0.],
[0., 1., 1., 1., 1., 1., 0., 0.],
[0., 1., 1., 1., 1., 1., 0., 0.],
[0., 1., 1., 1., 1., 1., 0., 0.]])
Or almost the same with columns:
torch.nn.functional.pad(example, (0, 0, 2, 1))
tensor([[0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0.],
[1., 1., 1., 1., 1.],
[1., 1., 1., 1., 1.],
[1., 1., 1., 1., 1.],
[1., 1., 1., 1., 1.],
[1., 1., 1., 1., 1.],
[0., 0., 0., 0., 0.]])