• Without being concerned so much about error checking and robustness, we can write something very simple to emulate the wget's core functionality.
• Full program is right here.
• It'll allow me to illustrate the most basic parts of the HTTP protocol, which are key to the assignment being released today.
• I'll run /usr/bin/wget (the built-in) and web-get (which is what we'll write) side by side in lecture.
• main entry point and implementation are presented here:

static const string kProtocolPrefix = "http://";
static const string kDefaultPath = "/";
static pair<string, string> parseURL(string url) {
  if (startsWith(url, kProtocolPrefix)) // in "string-utils.h"
    url = url.substr(kProtocolPrefix.size());
  size_t found = url.find('/');
  if (found == string::npos)
    return make_pair(url, kDefaultPath); // defined in <utility>
  string host = url.substr(0, found);
  string path = url.substr(found);
  return make_pair(host, path);
}

int main(int argc, char *argv[]) {
  if (argc != 2) {
    cerr << "Usage: " << argv[0] << " <url>" << endl;
    return kWrongArgumentCount;
  }
  pullContent(parseURL(argv[1]));
  return 0;
}     

• The parseURL function is a gesture to the programmatic split at the host-path border. In practice, more protocols (https, for instance) might be supported as well.

• We've already used createClientSocket function for our time-client program. Again, we'll conver its implementation and the implementation of createServerSocket on Wednesday during class.
• There is, of course, web-get-specific work to be done:

static void pullContent(const pair<string, string>& components) {
  int clientSocket = createClientSocket(components.first, 80);
  // error checking omitted    
  sockbuf sb(clientSocket);
  iosockstream ss(&sb);
  issueRequest(ss, components.first, components.second);
  skipHeader(ss);
  savePayload(ss, getFileName(components.second));
}

• The implementations of issueRequest, skipHeader, and savePayload subdivide the client-server conversation into manageable chunks.
• The best takeaway from the above is the fact that, for the second time in two examples, we've layered an iosockstream on top of a sockbuf, which itself is layered on top of our bidirectional socket descriptor.
  • Be glad we have the socket++ library, because without it, we'd need to do so much manual character array manipulation that coding would cease to be fun.
  • One very relevant piece of information about the sockbuf class: its destructor closes the file descriptor passed to it when it's constructed, so we shouldn't call close on clientSocket ourselves.

• Here's the implementation of issueRequest. Notice that I manually construct the absolute minimum, two-line request imaginable and send it over the wire to the server.
• It's standard HTTP-protocol practice that each line, including the blank line that marks the end of the request, end in CRLF (short for carriage-return- line-feed), which is '\r' following by '\n'.
• The flush call is necessary to ensure all character data is pressed over the wire and consumable at the other end.

static void issueRequest(iosockstream& ss, const string& host, const string& path) {
  ss << "GET " << path << " HTTP/1.0\r\n";
  ss << "Host: " << host << "\r\n";
  ss << "\r\n";
  ss.flush();
}

• After the flush, the client transitions from speak to listen mode.
  • The iosockstream is read/write, because the socket descriptor backing it is bidirectional.
• We read in all of the HTTP response header lines until we get to either a blank line or one that contains nothing other than a '\r'.
• The blank line is, indeed, supposed to be "\r\n", but some servers are sloppy, so we're supposed to treat the '\r' as optional. (Recall that getline chews up the '\n', but it'll leave '\r' at the end of the line).

static void skipHeader(iosockstream& ss) {
  string line;
  do {
    getline(ss, line);
  } while (!line.empty() && line != "\r");
}

• This is a reasonably legitimate situation where the do/while loop is the correct idiom. #yaydowhileloops

• Everything beyond the response header and that blank line is considered payload—that's the file, the JSON, the HTML, the image, the cat video.
• Every single byte that comes through should be replicated, in order, to a local copy.

static string getFileName(const string& path) {
  if (path.empty() || path[path.size() - 1] == '/') {
    return "index.html"; // not always correct, but not the point
  }

  size_t found = path.rfind('/');
  return path.substr(found + 1);
}

static const size_t kBufferSize = 1024; // just a random, large size
static void savePayload(iosockstream& ss, const string& filename) {
  ofstream output(filename, ios::binary); // don't assume it's text
  size_t totalBytes = 0;
  while (!ss.fail()) {
    char buffer[kBufferSize] = {'\0'};
    ss.read(buffer, sizeof(buffer));
    totalBytes += ss.gcount();
    output.write(buffer, ss.gcount());
  }
  cout << "Total number of bytes fetched: " << totalBytes << endl;
}

• The HTTP/1.0 protocol dictates that everything beyond that blank line is payload, and that once the server publishes each and every byte of the payload, it closes it's end of the connection. That server-side close is the client-side's EOF, and we write everything we read.
  • Note that we open an ofstream in binary mode, predominantly so the ofstream doesn't do anything funky with byte characters that are incidentally newline characters.
  • gcount returns the number of bytes read by the most recent call to read. (This I did not know until I wrote this example).
• The HTTP/1.1 protocol allows for connections to remain open, even after the initial payload has come through. I specifically avoid this here by going with HTTP/1.0, which doesn't allow this.