Linear Algebra

Linear algebra is the foundation of machine learning and deep learning, providing tools to understand and manipulate data in multi-dimensional spaces. This tutorial will introduce key concepts of linear algebra with Python examples using libraries like numpy, pandas, and matplotlib.

---

1. Linear Equations

Mathematical Description

A linear equation is an equation that represents a straight-line relationship. For example, the equation 2x+y=8,2x + y = 8 is linear. A system of linear equations consists of multiple such equations, and the goal is to find values of variables (like x,y,zx, y, z) that satisfy all equations simultaneously. These systems can be represented using matrices as:

Ax = b

Where:

• A: Matrix of coefficients.

• x: Vector of variables to be solved.

• b: Vector of constants.

Python Examples

Example 1: Solve a System of Linear Equations

Solve:

2x+y=8

x−y=2

import numpy as np

# Define the coefficient matrix and constants
A = np.array([[2, 1], [1, -1]])
b = np.array([8, 2])

# Solve for x
x = np.linalg.solve(A, b)
print("Solution: x =", x[0], ", y =", x[1])

Example 2: Visualizing Linear Equations

import matplotlib.pyplot as plt
import numpy as np

# Define the equations: y = 8 - 2x and y = x - 2
x = np.linspace(0, 5, 100)
y1 = 8 - 2 * x
y2 = x - 2

# Plot the equations
plt.plot(x, y1, label="2x + y = 8")
plt.plot(x, y2, label="x - y = 2")
plt.axhline(0, color='black',linewidth=0.5)
plt.axvline(0, color='black',linewidth=0.5)
plt.grid(color = 'gray', linestyle = '--', linewidth = 0.5)
plt.legend()
plt.title("Linear Equations")
plt.xlabel("x")
plt.ylabel("y")
plt.show()

---

2. Matrix Operations

Mathematical Description

A matrix is a rectangular array of numbers. Matrices are used to represent data, transformations, or systems of equations. Common operations include:

1. Addition: Add corresponding elements.

2. Multiplication: Multiply rows of one matrix by columns of another.

3. Transpose: Flip rows and columns.

Python Examples

Example 1: Matrix Addition and Multiplication

import numpy as np

# Define two matrices
A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])

# Addition
C = A + B

# Multiplication
D = np.dot(A, B)

print("Matrix Addition:\n", C)
print("Matrix Multiplication:\n", D)

Example 2: Transpose and Element-Wise Operations

# Transpose
transpose_A = A.T

# Element-wise multiplication
element_wise = A * B

print("Transpose of A:\n", transpose_A)
print("Element-wise multiplication:\n", element_wise)

---

3. Vector Spaces and Subspaces

Mathematical Description

A vector space is a set of vectors that can be added together or scaled. Subspaces are smaller vector spaces within a larger one, such as planes or lines passing through the origin. Linear independence of vectors determines if vectors span a space uniquely.

Python Examples

Example 1: Checking Linear Independence

# Define vectors
v1 = np.array([1, 2, 3])
v2 = np.array([4, 5, 6])
v3 = np.array([7, 8, 9])

# Form a matrix with the vectors
matrix = np.array([v1, v2, v3])

# Check the rank of the matrix
rank = np.linalg.matrix_rank(matrix)
print("Rank of the matrix:", rank)

if rank == len(v1):
    print("Vectors are linearly independent.")
else:
    print("Vectors are linearly dependent.")

Example 2: Visualizing a Subspace

import matplotlib.pyplot as plt

# Define basis vectors for a 2D subspace
v1 = np.array([1, 0])
v2 = np.array([0, 1])

# Plot vectors
plt.quiver(0, 0, v1[0], v1[1], angles='xy', scale_units='xy', scale=1, color='r', label='v1')
plt.quiver(0, 0, v2[0], v2[1], angles='xy', scale_units='xy', scale=1, color='b', label='v2')

plt.xlim(-1, 2)
plt.ylim(-1, 2)
plt.axhline(0, color='black',linewidth=0.5)
plt.axvline(0, color='black',linewidth=0.5)
plt.grid(color = 'gray', linestyle = '--', linewidth = 0.5)
plt.legend()
plt.title("2D Subspace Spanned by Vectors")
plt.show()

---

4. Linear Maps

Mathematical Description

A linear map (or transformation) takes vectors from one space to another while preserving addition and scalar multiplication. It can often be represented as matrix multiplication:

T(v) = Av

Python Examples

Example 1: Applying a Linear Transformation

# Define transformation matrix
T = np.array([[2, 0], [0, 3]])

# Define a vector
v = np.array([1, 1])

# Apply the transformation
result = np.dot(T, v)
print("Transformed vector:", result)

Example 2: Visualizing a Linear Transformation

# Original vectors
v1 = np.array([1, 0])
v2 = np.array([0, 1])

# Transformation matrix
T = np.array([[2, 1], [1, 2]])

# Transformed vectors
tv1 = np.dot(T, v1)
tv2 = np.dot(T, v2)

# Plot original and transformed vectors
plt.quiver(0, 0, v1[0], v1[1], color='blue', angles='xy', scale_units='xy', scale=1, label='v1 (original)')
plt.quiver(0, 0, v2[0], v2[1], color='green', angles='xy', scale_units='xy', scale=1, label='v2 (original)')
plt.quiver(0, 0, tv1[0], tv1[1], color='red', angles='xy', scale_units='xy', scale=1, label='T(v1)')
plt.quiver(0, 0, tv2[0], tv2[1], color='orange', angles='xy', scale_units='xy', scale=1, label='T(v2)')

plt.xlim(-1, 3)
plt.ylim(-1, 3)
plt.axhline(0, color='black',linewidth=0.5)
plt.axvline(0, color='black',linewidth=0.5)
plt.grid(color = 'gray', linestyle = '--', linewidth = 0.5)
plt.legend()
plt.title("Linear Transformation")
plt.show()

---

5. Matrix Decompositions

Mathematical Description

Matrix decomposition breaks a matrix into simpler matrices, making it easier to solve systems, compress data, or understand transformations. Common decompositions include:

1. Eigen Decomposition: A=QΛQ−1,A = Q\Lambda Q^{-1}, where Λ\Lambda contains eigenvalues.

2. Singular Value Decomposition (SVD): A=UΣVTA = U\Sigma V^T, useful in dimensionality reduction.

Python Examples

Example 1: Eigenvalues and Eigenvectors

# Define a matrix
A = np.array([[4, 2], [1, 3]])

# Compute eigenvalues and eigenvectors
eigenvalues, eigenvectors = np.linalg.eig(A)
print("Eigenvalues:", eigenvalues)
print("Eigenvectors:\n", eigenvectors)

Example 2: Singular Value Decomposition (SVD)

# Define a matrix
A = np.array([[1, 2], [3, 4], [5, 6]])

# Perform SVD
U, S, Vt = np.linalg.svd(A)
print("U:\n", U)
print("Singular values:", S)
print("V^T:\n", Vt)

---

This tutorial combines simple mathematical explanations with practical Python examples, equipping you with tools to tackle machine learning and deep learning problems.