Lecture Notes: Order Notation

Preliminaries

• Name tags
• Sign in

Course Content

Review 1

Experiment with the Pizza Demo. See how many combinations you can show before the website becomes unresponsive.

Review 2

Draw these functions:

• Constant
• Logarithmic
• Linear
• Quadratic
• Cubic
• Exponential

Review 3

Give four examples of functions which are \( O(n) \).

Motivation

• How long will an algorithm take to run?
• Let’s see a simple example:
• Pizza Demo
• A different example. Recall that: \( \sum_{i=1}^{n} i = \frac{n(n+1)}{2} \)
  • Clearly it will be faster to use the closed-form solution than the summation
• “Computational complexity” is an important topic in Computer Science, and you will encounter it many CS courses
• This week, we will cover the math behind computational complexity

Time and Memory

• We will analyze how the time needed to run an algorithm varies based on the size of the algorithm’s input
• We will focus on time, but memory usage is also important: some algorithms require less time because they use more memory.
• Note that algorithms use at least as much time as they do memory, since memory access requires time.
• Time can be measured using the computer’s clock, but when we analyze an algorithm we focus on the step count. Otherwise, our analysis would get outdated as newer, faster computers are released. Often, a line of code will correspond to a step. For example:
  • n = n + 1
  • z = x * y
• However, it is possible to write lines of code that perform many steps. For example:
  • sum(range(100))
  • filter(lambda x: x % 2 == 0, range(100))
• Steps can also be counted in terms of machine instructions, which you will see in CSCI 140: Assembly Language And Computer Organization.

Definitions

• Write: Definition: An algorithm is a fully specified computational method for producing outputs from inputs. The relation between an algorithm’s input and output is a function: uniquely defined for all values in the domain.
• Given enough time, we assume an algorithm will eventually stop computing.
• Write: Definition: An algorithm has runtime function \( f: \mathbb{N} \rightarrow \mathbb{N} \) if \( f(n) \) is the maximum number of steps the algorithm takes on any input of of size \( n \). This is a worst-case view of runtime.
• Worst-case analyses are useful, because then you don’t need to make assumptions about the kinds of inputs you will receive. For example, an attacker could send your server malicious inputs designed to waste the server’s resources (a denial of service attack).
• If you do know what kinds of inputs you will receive, you could perform an average-case analysis. But that won’t be our focus.

Example Runtime Functions

• Many algorithms have familiar runtime functions. Common functions:
  • Constant: \( f(n) = c \) for constant \( c \) not depending on \( n \)
  • Linear: \( f(n) = a \cdot n + b \) for constants \( a, b \)
  • Quadratic: \( f(n) = a \cdot n^2 + b \cdot n + c \) for constants \( a, b, c \)
  • Polynomial: \( f(n) = \sum_{i=0}^d a_i n^i \) for constants \( a_0, \cdots, a_d \)
    • If \( a_d > 0 \), then f has degree d.
    • Constant functions are 0-degree polynomials.
    • Linear functions are 1-degree polynomials.
    • Quadratic functions are 2-degree polynomials.
    • Cubic functions are 3-degree polynomials.
    • Non-integer degrees are also possible: fractional power functions.
  • Logarithmic: \( f(n) = \log_c(n) \) for constant \( c \)
  • Exponential: \( f(n) = c^n \) for constant \( c > 1 \), the base of the exponential function.
• We assume that runtime functions have domain and codomain \( \mathbb{N} \), so all values are nonnegative
• For all except constant functions, we assume the larger the input size, the longer the runtime.
• Write: Definition: A function \( f \) is monotonically increasing if \( m ≤ n \) then \( f(m) ≤ f(n) \).
• Let’s consider the runtime of some simple algorithms:
  • Constant: add two numbers
  • Logarithmic: look up a term in an encyclopedia using binary search
  • Linear: Sum a series of \( n \) numbers
  • Quadratic: Initialize a matrix of size \( n \times n \)
  • Exponential: Print all subsets of \( n \) available pizza toppings

Input Size

• What we consider the size of an algorithm’s input depends on how the input is used.
• Suppose a program accepts a string as input.
• Ask: In each of these cases, what is the runtime of the algorithm, and what is \( n \)?
  • A sequence of five integers, which will be averaged: my.py 1 3 5 7 9
    • Constant
  • A sequence of integers of unlimited length, which will be averaged: my.py 1 3 5 7 9 ...
    • Linear, where \( n \) is the number of integers
  • One integer, which describes the dimensions of a randomly initialized matrix: my.py 9
    • Quadratic, where \( n \) is the integer
  • A sequence of ingredients of unlimited length, which will be used to form all possible combinations of ingredients: my.py cheese tomatoe pepperoni ...
    • Exponential, where \( n \) is the number of ingredients

Comparing Functions

• We can compare functions by forming a ratio.
• For example, if \( f(n) = n \) and \( g(n) = 2n \) then: \[ \frac{f(n)}{g(n)} = \frac{n}{2n} = \frac{1}{2} \]
• The ratio can also depend on the input size.
• For example, consider: \( f(n) = n + 2 \) and \( g(n) = 2n + 1 \).
• If \( n = 1 \): \[ \frac{f(n)}{g(n)} = \frac{1 + 2}{2 + 1} = 1 \]
• But if \( n = 100 \): \[ \frac{f(n)}{g(n)} = \frac{100 + 2}{200 + 1} \approx 0.51 \]
• We are most intersted in the ratio when the input \( n \) is large.
• Suppose \( f \) and \( g \) are runtime functions.
• When \( n \) is small, the runtimes are close.
• But as \( n \) grows larger, is becomes clear that that algorithm with runtime \( f \) is much faster!
• Write: Definition: We will analyze the limit of the ratio as \( n \) approaches infinity, written \( \lim_{n \to \infty} \)
• Write: Basic limits: \[ \lim_{n \to \infty} c = c \text{ for any constant } c \] \[ \lim_{n \to \infty} n = \infty \] \[ \lim_{n \to \infty} \frac{1}{n} = 0 \]
• Write: Infinity arithmetic: \[ \infty + \infty = \infty \cdot \infty = \infty \] \[ 3 + \infty = 3 \cdot \infty = \infty \]
• Write: Theorem: For any nonnegative functions \( f(n) \) and \( g(n) \), \[ \lim_{n \to \infty} \big( f(n) + g(n) \big) = \big( \lim_{n \to \infty} f(n) \big) + \big( \lim_{n \to \infty} g(n) \big) \]
• More rules are mentioned in the textbook’s Theorem 21.2.

Asymptotic Equivalence

• Typically, we want to approximate the runtime of algorithms.
• For our purposes, it is sufficient to find a simple function which is equivalent to the algorithm’s runtime.
• Write: Definition: Two functions \( f(n) \) and \( g(n) \) are said to be asymptotically equivalent, denoted \( f(n) \sim g(n) \), if and only if \[ \lim_{n \to \infty} \frac{f(n)}{g(n)} = 1 \]
• This means that lower-order terms can be ignored. For example, consider \( f(n) = n^2 + n + 1 \) and \( g(n) = n^2 \). \[ \lim_{n \to \infty} \frac{n^2 + n + 1}{n^2} = 1 + \lim_{n \to \infty} \frac{1}{n} + \lim_{n \to \infty} \frac{1}{n^2} = 1 \]
• Thus, \( f(n) \sim g(n) \)
• It isn’t necessary to name the runtime functions: we could simply write: \( n^2 + n + 1 \sim n^2 \)

Growth Order

• Actually, it is beneficial to relax things further.
• For example, \( 2n \not\sim n \). However, it might be useful to consider these runtimes as roughly equivalent.
• As processors get faster, programs often run faster by a constant factor.
• If we compare functions by how quickly they grow, we can treat functions that differ asymptotically by only a constant factor as roughly equivalent.
• Write: Definition: The growth rate of a function is referred to as its order.
• We will see three different ways of describing growth order:
  • Big O, \( O \), upper bound
  • Big Theta, \( \Theta \), tight bound
  • Big Omega, \( \Omega \), lower bound

Big O

• Write: Definition: \( O(g(n)) \) denotes the set of functions that asymptotically grow no faster than \( g(n) \).
• Mathematically, \( f(n) \in O(g(n)) \) if and only if there exists a constant \( c \) and a minimum value \( n_0 \) such that for all \( n \geq n_0 \), \[ f(n) \leq c \cdot g(n) \]
• Or equivalently, \( f(n) \in O(g(n)) \) if and only if: \[ \lim_{n \to \infty} \frac{f(n)}{g(n)} < \infty \]
• Draw: \( f(x) = a \cdot x \) and \( g(x) = b \cdot x^2 \). Show that no matter the coefficients, as \( n \) grows large, the second function will outgrow the first.
• Ask: Let’s list some other functions which are \( O(n^2) \).
• Answer: \( 2, 2n + 1, 2n^2, \log(n) \)
• Ask: How about functions which are \( O(n^3) \)?

Big Omega

• Big O was the upper bound. We can also describe a lower bound, which we call Big Omega.
• Write: Definition: \( \Omega(g(n)) \) denotes the set of functions that asymptotically grow no more slowly than \( g(n) \).
• Mathematically, \( f(n) \in \Omega(g(n)) \) if and only if there exists a constant \( c \) and a minimum value \( n_0 \) such that for all \( n \geq n_0 \), \[ f(n) \geq c \cdot g(n) \]
• Or equivalently, \( f(n) \in \Omega(g(n)) \) if and only if: \[ \lim_{n \to \infty} \frac{f(n)}{g(n)} > 0 \]
• Ask: Let’s list some functions which are \( \Omega(n^2) \).
• Answer: \( 2n^2, 0.5n^2, n^3, 2^n \)
• Write: Theorem: \( f(n) \in O(g(n)) \) if and only if \( g(n) \in \Omega(f(n)) \)

Big Theta

• Finally, we can also describe a tight bound, which we call Big Theta.
• Write: Definition: \( \Theta(g(n)) \) denotes the set of functions that asymptotically grow at the same rate as \( g(n) \). If \( f(n) \in O(g(n)) \) and \( f(n) \in \Omega(g(n)) \), then \( f(n) \in \Theta(g(n)) \).
• Mathematically, for some constant \( c \), where \( 0 < c < \infty \), \( f(n) \in \Theta(g(n)) \) if and only if: \[ \lim_{n \to \infty} \frac{f(n)}{g(n)} = c \]
• Ask: Let’s list some functions which are \( \Theta(n^2) \).
• Answer: \( 2n^2, 0.5n^2, n^2 + n + 1 \)

Summary

\[ O(c) \subsetneq O(\log(n)) \subsetneq O(n) \subsetneq O(n^2) \subsetneq O(n^3) \subsetneq O(2^n) \] \[ \Omega(2^n) \subsetneq \Omega(n^3) \subsetneq \Omega(n^2) \subsetneq \Omega(n) \subsetneq \Omega(\log(n)) \subsetneq \Omega(c) \] \[ \Theta(n^2) = \Theta(2n^2) = \Theta(2n^2 + 2n + 2) \] \[ \Theta(n) = \Theta(2n) = \Theta(2n + 2) \]

Little O and Little Omega

• It is also useful to describe when functions grow slower or faster than other functions. We use “little o” and “little omega” for this purpose.
• Write: Definition: \( o(g(n)) \) denotes the set of functions that asymptotically grow slower than \( g(n) \).
• Mathematically, \( f(n) \in o(g(n)) \) if and only if: \[ \lim_{n \to \infty} \frac{f(n)}{g(n)} = 0 \]
• For example, \( n \in o(n^2) \). Whereas \( n^2 \not\in o(n^2) \).
• Write: Definition: \( \omega(g(n)) \) denotes the set of functions that asymptotically grow faster than \( g(n) \).
• Mathematically, \( f(n) \in \omega(g(n)) \) if and only if: \[ \lim_{n \to \infty} \frac{f(n)}{g(n)} = \infty \]
• For example, \( n^2 \in \omega(n) \). Whereas \( n^2 \not\in \omega(n^2) \).

Theorems

• When analyzing growth order, we focus on the highest order term. These theorems formalize that. See textbook for proofs.
• Write: Theorem: Suppose \( f(n) \) is a polynomial of degree \( a \), and \( g(n) \) is a polynomial of degree \( b \).
  • If \( a = b \), then \( f(n) \in \Theta(g(n)) \)
  • If \( a \leq b \), then \( f(n) \in O(g(n)) \)
  • If \( a \geq b \), then \( f(n) \in \Omega(g(n)) \)
• Write: Theorem: If \( f(n) = \log_c(n) \) and \( g(n) = \log_d(n) \) for constants \( c, d > 1 \), then \( f(n) \in \Theta(g(n)) \).
• Write: Theorem: For a logarithmic function \( f(n) = \log_c(n) \) and a polynomial function \( g(n) = c_0 + c_1 n^1 + c_2 n^2 + \ldots + c_i n^i \), \( f(n) \in o(g(n)) \).

Example Algorithms

• Draw: The data structure for each of the following.
• Ask: What is the runtime of each of these algorithms?
  • Initialize a matrix of size \( n \times n \times n \). Answer: Cubic.
  • Sum the elements of an array of length \( n \). Answer: Linear.
  • Initialize 10 matrices of size \( n \times n \). Answer: Quadratic.
  • Locate an element in an ordered array of length \( n \). Answer: Logarithmic.
  • Print all subsets of \( n \) available pizza toppings. Answer: Exponential.
• Suppose that for \( n = 10 \), each algorithm takes 10ms to run.
• Ask: How long do we expect the algorithm to run if \( n = 20 \)?
  • Linear: 20ms
  • Quadratic: 40ms
  • Cubic: 80ms
  • Logarithmic: ≈13ms
  • Exponential: ≈10,000ms
• Note that these are just estimates, since we are ignoring lower-order terms. To get a better estimate, you could plot the runtime for a few different input sizes.

Connection to Counting

• The number of possible pizzas depends on the available toppings. For each topping, you must choose whether to include it or not. So \( 2^n \) different pizzas are possible.
• This is an example of counting, which will explore more next week.
• Let’s see a similar example: design your own burger.
• Draw: Choices:
  • Cook time: Rare, Medium, Well-Done
  • Toppings: Lettuce, Tomato, Onions, Ketchup, Mayo, Mustard
• Here we have three choices for cook time, and two choices for each topping.
• So there are: \( 3 \cdot 2^6 = 192 \) possible burgers.

Compute Budget

The textbook works through an example of compute budget on page 228. Skipped for time.