Today: modules, how the internet works

Today: "import"

There are a few lines in your Python files that still need to be explained. Today we'll explain the "import" lines, e.g in pylibs.py.

...

import sys
import random

def read_terms(filename):
...

Your Code On Python

Your Python code runs "on" the Python system installed on your computer, which we can think of as a sort of stack like this:

Python code is "portable", meaning that you can move your Python code, without any editing, to a different type of computer or operating system, and it should run there too. This is a nice feature, allowing your Python code work reliably over time.

import math - Module - Library

• This line imports a module named "math"
• The math module contains functions ready to use
• The import line brings that module into Python so your code can use it
• In particular, the math module contains common mathematical functions like sin(), cos()
• Also known as a "library" of code

Modules and Modern Coding

Python itself has the basic loop/string etc functionality. And there are modules that contain ready to use code for common problems. e.g. the "math" module contains common math functions, the "urllib" module contains code to interact with urls and the web.

We say that you build your code "on top of" the module code. Modern coding is part custom code, and part building on top of module code.

Easy To ♥ Modules

• Question while coding:
  Is the code for this solved in some module already?
• Using module code you didn't have to write is very attractive
• This is kind of a no-brainer case to make!
• Somebody else wrote it, you can just use it
• It's well tested
• It has real documentation - aka docs
• Your teammates may already be familiar with it
• CS106A we see this "module" theme a little
  CS106A needs to cover fundamentals
  loops, strs, dicts, files ..
• Courses beyond CS106A, you will likely use modules more

Module = Name + Code + Docs

• A module has a name, e.g. "math"
• A module contains functions, solving common problems
• A module also has documentation "docs" explaining the use of its functions
• e.g. "math" module contains math functions
• e.g. "random" module contains functions for pseudo-random numbers
• e.g. "urllib" module contains functions for urls and web requests

Step 1: import math

• To use a module, include a import math line
• Import the module by its name
• Typically these are grouped near the top of your Python file

Step 2: math.sqrt(2)

• On later lines, refer to functions in the module with a dot
• e.g. math.sqrt(2)
• e.g. random.randrange(10)
• Readable: in this way, it's clear when calling a function..
  that it is coming from that module
• There are other ways of doing import
• This form is simplest: import math .. math.sqrt()

>>> import math
>>> math.sqrt(2)  # call sqrt() fn
1.4142135623730951
>>> math.sqrt
<built-in function sqrt>
>>> 
>>> math.log(10)
2.302585092994046
>>> math.pi       # constants in module too
3.141592653589793

Common Error: Missing Import

Quit and restart the interpreter without the import, see common error:

>>> math.sqrt(2)  # OOPS forgot the import
Traceback (most recent call last):
NameError: name 'math' is not defined
>>>
>>> import math
>>> math.sqrt(2)  # now it works
1.4142135623730951

Module = Dependency

• When you write code using a module
• Your code now depends on that module's existence
• If that module disappeared, your code would stop working

1. "Standard" Modules — Fine

• Standard = included/maintained as part of Python3 install
• e.g. math, sys, datetime
• Use without hesitation, can rely on these modules now and in the future
  Very rare for a module to be dropped
• No separate install is required
• All the standard modules are installed when python itself is installed

Many Standard Modules

• Do not need to memorize whole list of modules
• Do check list for help when starting a project
• Standard Python Modules List
• A few examples...
• csv module to read CSV files, such as produced by Excel
• datetime module of calendar functions
• zipfile module for reading/creating .zip files
• urllib module for making web http requests from Python

2. Non-Standard "pip" Modules — Tricky

Other modules are valuable but they are not a standard part of Python. For code using non-standard module to work, the module must be installed on that computer via the "pip" Python tool. e.g. for homeworks we had you pip-install the "Pillow" module with this command:

$ python3 -m pip install Pillow
..prints stuff...
Successfully installed Pillow-5.4.1

A non-standard module can be great, although the risk is harder to measure. The history thus far is that popular modules continue to be maintained. Sometimes the maintenance is picked up by a different group than the original module author. A little used module is more risky.

Upgrade Python -> Re-Install Pip Modules

When you upgrade Python, from 3.11 to to 3.12, you will lose the pip installed modules which are back in your previous Python 3.11 directories. You need to re-install your pop modules - not hard actually.

Security: Module vs. Supply Chain Attack

1. Module Requires Trust

When you install a module on your machine from somewhere - you are trusting that code to run on your machine. In very rare cases, bad guys have tampered with modules to include malware in the module, which then runs on your machine, steal data, install malware, etc. A so called "supply chain attack"

2. Well Known / Safe: python.org, Pillow, matplotlib

Installing code from python.org is very safe, and also very well known modules like Pillow and matplotlib are very safe, benefiting from large, active base of users.

3. Problems: pypi.org, matplotplib typosquatting

Several supply chain attacks have been made on lesser known modules, from lesser known code sources, in particular the code source pypi.org. They are aware of this problem and are working on it. There are also "typosquatting" attacks, where bad guys put up a malware module with a name easily confused with the real module, something like a "matplotplib", expecting that some unwary people will install the bad one by accident.

Be more careful if installing a little used module. Prefer code that was released a month ago vs. code that was released yesterday, allowing time for the community to notice if something is not right.

Module Docs

• Every module has formal "documentation" - "docs"
• Explain what its functions do
• Remember abstraction vs. implementation
• The docs describe the abstractions of the functions
  What each function does
  How to call it
• Demo web search: "python math module"
• python.org - the official home of python docs
  Official python.org math docs
  Watch out for SEO
  Search Engine Optimization
  Some possibly lame site tries to get a better search ranking

Hacker: Use dir() and help() (optional)

• Feel like a hacker, use dir() and help() on module
• In the interpreter >>>
• dir(module) - shows a list of all the defs in the module
• help(module.fn) - shows some help text for that function
• The """Pydoc""" we write to describe each function
• That Pydoc is what help() returns (demo later)
• Names starting with double-underbar = internal thing you should probably not mess with

>>> import math
>>> dir(math)
['__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'acos', 'acosh', 'asin', 'asinh', 'atan', 'atan2', 'atanh', 'ceil', 'copysign', 'cos', 'cosh', 'degrees', 'e', 'erf', 'erfc', 'exp', 'expm1', 'fabs', 'factorial', 'floor', 'fmod', 'frexp', 'fsum', 'gamma', 'gcd', 'hypot', 'inf', 'isclose', 'isfinite', 'isinf', 'isnan', 'ldexp', 'lgamma', 'log', 'log10', 'log1p', 'log2', 'modf', 'nan', 'pi', 'pow', 'radians', 'remainder', 'sin', 'sinh', 'sqrt', 'tan', 'tanh', 'tau', 'trunc']
>>>
>>> help(math.sqrt)
Help on built-in function sqrt in module math:

sqrt(x, /)
    Return the square root of x.
>>>
>>> help(math.cos)
Help on built-in function cos in module math:

cos(x, /)
    Return the cosine of x (measured in radians).

How to Create Your Own Module?

How hard is it to write a module? Not hard at all. A regular file like wordcount.py is also a module.

Consider the file wordcount.py in wordcount.zip

Forms a module named wordcount

• Suppose you have built some useful functions
• Someone else in your lab wants to use them....
  Them pasting in their own copy is not ideal
• We have wordcount.py
• python3 wordcount.py - runs main()
• wordcount.py is also a module named just "wordcount"
• Think of all the defs in wordcount: read_counts(), clean(), print_counts(),
• import works on wordcount (in the same directory)
• Access functions as module.xxx just like usual
• Run python interpreter in wordcount directory to try this
• Try importing wordcount, calling the read_counts() function
• Or call wordcount.clean()

Try this demo in the wordcount directory

>>> # Run interpreter in wordcount directory
>>> import wordcount
>>>
>>> wordcount.read_counts('poem.txt')
{'roses': 1, 'are': 2, 'red': 1, 'violets': 1, 'blue': 1, 'this': 1, 'does': 1, 'not': 1, 'rhyme': 1}
>>>

How babygraphics.py Used babynames.py

# 1. In the babygraphics.py file
# import the babynames.py file in same directory
import babynames

...

    # 2. Call the read_files() function                                                                  
    names = babynames.read_files(FILENAMES)

Style: Good Decomposition = Good Re-Use

• We write functions with black-box decomposition
  Function solves one problem
  Takes in data as parameters
  Returns result data
  e.g.read_counts(filename)
• This structure is also good for re-use across modules
• The function is not hard-coded to just one situation
• The function works in future situations, using whatever new data is passed-in
• A form of generality - the function can be used in new situations

dir() and help() work on wordcount Too

• Look at wordcount.py, look at the functions
• dir() and help() work here too
• See where the """Pydoc""" goes!

>>> dir(wordcount)
['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'clean', 'main', 'print_counts', 'print_top', 'read_counts', 'sys']
>>> 
>>> help(wordcount.read_counts)

read_counts(filename)
    Given filename, reads its text, splits it into words.
    Returns a "counts" dict where each word
    ...

Module Example: urllib

• Look at how web pages work
• Then play with urllib

How Does The Web Work?

• Web browser app has a url (the "client" side)
• Web server is running on some machine (the "server" side)
• The browser sends a reguest to the server over the internet
• The server sends back a response data, often HTML, describing the page to show for that url
• HTML is a text code that describes a page of words and pictures
• The browser "renders" the HTML on screen
• Common request/response data types: HTML JPEG PNG GIF SVG
• New open video format "AV1" .. you heard it here first!

HTML

Suppose we have this text on screen

This important text

Here is the HTML code to produce the above, with tags like <b> inserted in the text to indicate bold, paragraphs, urls etc. etc. sprinkled in the text..

This <b>important</b> text

Go to python.org. Try view-source command on this page (right click on page). Search for a word in the page text, such as "whether" .. to find that text in the HTML code.

Think of how many web pages you have looked at - this is the code behind those pages. It's a text format! Lines of unicode chars!

Web Page - HTML - Python

Every web page you've ever seen is defined by this HTML text behind the scenes. Hmm. Python is good at working with text.

urllib Demo

• Python code to request an HTML page by url
• urllib - making requests to a server, getting back data
• An example of using a standard module
• docs - urllib.request docs on python.org
• Makes a URL look like a local file mostly
• Read the text of the web page
• Use s.find() to display a fragment of it

(See copy of these lines below suitable for copy/paste yourself.)

>>> import urllib.request
>>> f = urllib.request.urlopen('http://www.python.org/')
>>> text = f.read().decode('utf-8')
>>> text.find('Whether')
26997
>>> text[26997:27100]
"Whether you're new to programming or an experienced developer, it's easy to learn and use Python"

Here is the above Python lines, suitable for copy paste:

import urllib.request
f = urllib.request.urlopen('http://www.python.org/')
text = f.read().decode('utf-8')

• f.read() - reads all the bytes in one call
• decode('utf-8') - decode raw bytes, returning unicode string
• Does not always work, they may be blocking python
• Also some web sites convey their HTML in a different way
• Can try http: or https:

What Just Happened

• Normally you have a url, enter it in your browser, see the web page
• Now a url works from within python
• urllib takes in a url, fetches its data into a Python variable

Data From the Web vs. Files

• Very often data is drawn from file, into your program
• New picture: data is drawn from a URL
• e.g. 100 temperature sensors, each with its own url, sample them from Python
• urllib packages up a few functions for this
  Handling the details
• Also a nice example of a module
  Packaging up the code to solve a common problem, you just call it

How Does The Internet work?

Just for fun, how does the Internet work? I pulled these slides together I had laying around from another class. Neat to see how something you use every day works.

Internet - TCP/IP Standards

• Internet - world-wide network built on open standards
• Open standards: TCP, IP, HTML, JPEG, PNG
• These are vendor neutral - unbelievably successful pattern
  -No license required
  -No patents required
  -Brings in huge participation
  -Huge participation can create great value
• Somehow free/open standard have created astronomical economic value
• Future economics thesis topic for someone

1. Packets

• Say we have 50KB image.jpg
• 50,000 bytes
• Divide the image into packets
• Say each packet is 1500 bytes (a common size)
• Networking does everything in terms of packets

Sending One Packet

• Simplified look at transmission of one packet
• Each byte is 8 bits, e.g. 0 1 1 0 1 0 1 0
  -Go through all the bits, from left to right
  -For each 1, put 3 volts on the wire
  -For each 0, put 0 volts on the wire
• Modern ethernet is runs at at least 100 million bits per second
  -aka 100 mbps
• In reality, there is more complex scheme than 3-volts/0-volts
• Wifi sends the 0/1 data on a radio channel
  -The encoding is more complex
  -But you get the idea

IP Address

• Every computer on the internet has an IP address
• Here looking at IP v4 addresses, v6 addrs some day in the future
• e.g. 171.64.64.166
• IP addr is exactly 4 bytes (4 1-byte numbers)
• Left part encodes "neighborhood" on internet
• Just like phone, 650-725-0000
• e.g. 171.64.xxx.xxx generally Stanford campus
• e.g. 171.64.64.xxx my floor of Gates building
• Cannot just make up an IP addr, depends on location

(optional) Sandra Bullock Blooper

TCP/IP blooper in this video of "The Net"...

video

Blooper: in the video the IP address is shown as 75.748.86.91 - not a valid IP address! Each number should be 1 byte, 0..255

Router

• A computer typically connects to a router for service
  -We'll say the router is "upstream"
• The router provides the internet service, handling the computer's packets
• Typically when you connect to wi-fi, you get a router too
• The router will assign a temporary IP address to the computer
• The router typically has the IP address ending with ".1"

The most common way for a computer to be "on the internet" is to establish a connection with a "router" which is already on the internet. The computer establishes a connection via, say, wifi to communicate packets with the router. The router is "upstream" of the computer, connecting the computer to the whole internet.

IP Packet - Hopping

• Say Netflix is sending a packet with 1/100th second of video to your laptop
• For a computer to send a packet:
• Label it with the IP address of the source and destination
• Send it to local router
• Router in turn sends it on to bigger router
• The packet proceeds .. hop hop hop .. over to final destination

Lots of Packet Hops

The packet is passed from router to router - called a "hop". There might be 10 or 20 hops in a typical internet connection.

Routers - Decentralized and Cooperative

• Each router knows enough to figure the next hop, not the whole route
• There is no "center" of the internet that knows everything
• The initiating computer does not typically know anything, delegating to its router
• "Core" routers, towards the middle, bigger, fancier, more connections
• Routers measure connection functionality/breakage all the time
  -Choose alternative routes in real time from breakage, congestion
  -e.g. a backhoe breaks an important link, routers re-route in realtime
• Routers are a distributed, cooperative system
  -Each does its part, cooperatively solving the whole thing
  -VS. centralized/top-down system
• Authoritarian countries, most famously China, control the routers within the country, blocking or tampering with certain traffic

The routing of a packet from your computer is like a capillary/artery system .. your computer is down at the capillary level, your packet gets forwarded up to larger and larger arteries, makes its way over to the right area, and then down to smaller and smaller capillaries again, finally arriving at its destination.

Note: Special "Local" IP Addresses

• Note that 10.x.x.x and 192.168.x.x addresses are special "local" IP addresses
• These addresses are not valid out on the internet at large
• They are used within an organization but not outside it
• These are translated to a real IP addr as a packet makes its way
• Frequently given out by Wi-Fi routers .. why I mention them

What Does it Mean to Be On the Internet?

• On the internet - e.g. connect to a Wi-Fi router
• 1. Computer connects to an upstream router to handle traffic. Most Wi-Fi access points combine Wi-Fi radios and a router.
• 2. The router typically gives the computer an IP address to use
• The computer cannot pick an arbitrary IP address, since the left part of the address depends on the location on the internet ... details known by the router
• Also, you don't want to pick an IP address in use by someone else, so the router gives you a known good one
• 3. DHCP "Dynamic Host Configuration Protocol" - automatically configure network settings to work locally. Computers very often use this feature to get needed network configuration from the router automatically.

So what does it mean for a computer to be on the internet? Typically it means the computer has established a connection with a router. The commonly used DHCP standard (Dynamic Host Configuration Protocol), facilitates connecting to a router; establishing a temporary connection, and the router gives your computer an IP address to use temporarily. Typically DHCP is used when you connect to a Wi-Fi access point.

Demo-1: Your Internet

Bring up the networking control panel of your computer. It should show what IP address you are currently using and the IP address of your router. You will probably see some text mentioning that DHCP is being used. Your computer will likely have a local IP address, just used while you're in this room.

Demo-2: Ping

"Ping" is an old and very simple internet utility. Your computer sends a "ping" packet to any computer on the internet, and the computer responds with a "ping" reply (not all computers respond to ping). In this way, you can check if the other computer is functioning and if the network path between you and it works. As a verb, "ping" is also used in regular English this way .. not sure if that's from the internet or the other way around.

Experiment: Most computers have a ping utility, or you can try "ping" on the command line (works on the Mac, Windows, and Linux). Try pinging www.google.com or pippy.stanford.edu. Not all computers respond to ping. Type ctrl-c to terminate ping.

Milliseconds fraction of a second used for the packet to go and come back. 1 ms = 1/1000 of a second. Different from bandwidth, this "round trip delay".

Here I run the "ping" program for a few addresses, see what it reports. Every second, a ping packet is sent and the result is printed.

$ ping www.google.com  # I type in a command here
PING www.l.google.com (74.125.224.144): 56 data bytes
64 bytes from 74.125.224.144: icmp_seq=0 ttl=53 time=8.219 ms
64 bytes from 74.125.224.144: icmp_seq=1 ttl=53 time=5.657 ms
64 bytes from 74.125.224.144: icmp_seq=2 ttl=53 time=5.825 ms
^C                            # Type ctrl-C to exit
--- www.l.google.com ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 5.657/6.567/8.219/1.170 ms
$ ping pippy.stanford.edu
PING pippy.stanford.edu (171.64.64.28): 56 data bytes
64 bytes from 171.64.64.28: icmp_seq=0 ttl=64 time=0.686 ms
64 bytes from 171.64.64.28: icmp_seq=1 ttl=64 time=0.640 ms
64 bytes from 171.64.64.28: icmp_seq=2 ttl=64 time=0.445 ms
64 bytes from 171.64.64.28: icmp_seq=3 ttl=64 time=0.498 ms
^C
--- pippy.stanford.edu ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.445/0.567/0.686/0.099 ms

Traceroute

• On the mac, just open a terminal as shown below
• How to do: traceroute on windows
• See series of hops
• First few hops in your IP neighborhood
• Farther away hops .. more milliseconds
• Note New York, London .. big jump in milliseconds
• Hops does not go up linearly with distance
• East bay - 30 miles from Stanford - 11 hops
• Russia - 10000 miles from Stanford - 19 hops

Traceroute is a program that will attempt to identify all the routers in between you and some other computer out on the internet - demonstrating the hop-hop-hop quality of the internet. Most computers have some sort of "traceroute" utility available if you want to try it yourself (not required). On Windows it's called "tracert" in Windows Power Shell, and it does not suppor the "-q 1" option below, but otherwise works fine.

Some routers are visible to traceroute and some not, so it does not provide completely reliable output. However, it is a neat reflection of the hop-hop-hop quality of the internet.s

Traceroute codingbat.com

codingbat.com is housed in the east bay - 13 hops we see here. The milliseconds listed is the round-trip delay.

$ traceroute -q 1 codingbat.com
traceroute to codingbat.com (173.255.219.70), 64 hops max, 52 byte packets
 1  rt-ac68u-b3f0 (192.168.1.1)  7.152 ms
 2  96.120.89.177 (96.120.89.177)  9.316 ms
 3  24.124.159.189 (24.124.159.189)  9.638 ms
 4  be-232-rar01.santaclara.ca.sfba.comcast.net (162.151.78.253)  9.775 ms
 5  be-39931-cs03.sunnyvale.ca.ibone.comcast.net (96.110.41.121)  31.753 ms
 6  be-3202-pe02.529bryant.ca.ibone.comcast.net (96.110.41.214)  10.273 ms
 7  ix-xe-0-1-1-0.tcore1.pdi-paloalto.as6453.net (66.198.127.33)  10.570 ms
 8  if-ae-2-2.tcore2.pdi-paloalto.as6453.net (66.198.127.2)  11.344 ms
 9  if-ae-5-2.tcore2.sqn-sanjose.as6453.net (64.86.21.1)  13.555 ms
10  if-ae-1-2.tcore1.sqn-sanjose.as6453.net (63.243.205.1)  11.583 ms
11  216.6.33.114 (216.6.33.114)  11.938 ms
12  if-2-4.csw6-fnc1.linode.com (173.230.159.87)  14.833 ms
13  li229-70.members.linode.com (173.255.219.70)  11.549 ms

Traceroute Serbia

A random Serbian address - 31 hops - the farthest thing I could fine. See the extra delay where the packets go across the Atlantic - I'm guessing hop 16. The names there may refer to Amsterdam and France. Note that the packets are going at a fraction of the speed of light here - a fundamental limit of how quickly you can get a packet across the earth.

Or try: www.fu-berlin.de or www.parlament.hu be1.rtr1.vh.hbone.hu

May hang at the end, but we can at least see the early hops.

$ traceroute -q 1 yujor.fon.bg.ac.rs
traceroute to hostweb.fon.bg.ac.rs (147.91.128.13), 64 hops max, 52 byte packets
 1  rt-ac68u-b3f0 (192.168.1.1)  9.136 ms
 2  96.120.89.177 (96.120.89.177)  9.608 ms
 3  24.124.159.189 (24.124.159.189)  20.184 ms
 4  be-232-rar01.santaclara.ca.sfba.comcast.net (162.151.78.253)  15.058 ms
 5  be-39911-cs01.sunnyvale.ca.ibone.comcast.net (96.110.41.113)  11.050 ms
 6  be-3411-pe11.529bryant.ca.ibone.comcast.net (96.110.33.94)  11.294 ms
 7  be3111.ccr31.sjc04.atlas.cogentco.com (154.54.11.5)  10.420 ms
 8  be2379.ccr21.sfo01.atlas.cogentco.com (154.54.42.157)  20.021 ms
 9  be3110.ccr32.slc01.atlas.cogentco.com (154.54.44.142)  37.200 ms
10  be3037.ccr21.den01.atlas.cogentco.com (154.54.41.146)  36.318 ms
11  be3035.ccr21.mci01.atlas.cogentco.com (154.54.5.90)  49.991 ms
12  be2831.ccr41.ord01.atlas.cogentco.com (154.54.42.166)  66.591 ms
13  be2718.ccr22.cle04.atlas.cogentco.com (154.54.7.130)  67.178 ms
14  be2993.ccr31.yyz02.atlas.cogentco.com (154.54.31.226)  77.369 ms
15  be3260.ccr22.ymq01.atlas.cogentco.com (154.54.42.90)  86.026 ms
16  be3042.ccr21.lpl01.atlas.cogentco.com (154.54.44.161)  152.559 ms
17  be2183.ccr42.ams03.atlas.cogentco.com (154.54.58.70)  161.324 ms
18  be2813.ccr41.fra03.atlas.cogentco.com (130.117.0.122)  164.945 ms
19  be2960.ccr22.muc03.atlas.cogentco.com (154.54.36.254)  172.507 ms
20  be2974.ccr51.vie01.atlas.cogentco.com (154.54.58.6)  197.670 ms
21  be3463.ccr22.bts01.atlas.cogentco.com (154.54.59.186)  181.075 ms
22  be3261.ccr31.bud01.atlas.cogentco.com (130.117.3.138)  184.336 ms
23  be2246.rcr51.b020664-1.bud01.atlas.cogentco.com (130.117.1.14)  189.231 ms
24  149.6.182.114 (149.6.182.114)  182.364 ms
25  amres-ias-amres-gw.bud.hu.geant.net (83.97.88.6)  191.607 ms
26  amres-mpls-core----amres-ip-core-amres-ip.amres.ac.rs (147.91.5.144)  187.181 ms
27  *
28  stanica-134-241.fon.bg.ac.rs (147.91.134.241)  204.945 ms
29  stanica-134-250.fon.bg.ac.rs (147.91.134.250)  192.673 ms
30  stanica-134-250.fon.bg.ac.rs (147.91.134.250)  193.978 ms
31  stanica-134-250.fon.bg.ac.rs (147.91.134.250)  193.032 ms !Z

TCP/IP Summary Picture