Understanding & Debugging Network Requests

I work on web applications for a living. I meander across the stack, but my experience tends towards front-end engineering. Legitimately nothing I do would be possible without HTTP requests.

Despite that, I found anything “network” — routing, middleware, cookies, headers etc. — pretty intimidating. When something went wrong with a network request, I quickly ran out of tools in my debugging tool kit. I wasn’t sure how to understand or explore network request issues in the applications I was working on.

I want to try to demystify some of this, and share what I’ve learned over the last couple of years.

This will be a high-level overview of the network layer along with deeper examinations of network requests, both in the browser and in a JavaScript application.

I’ll also explore some of the debugging tools provided by browsers, walk through a request being passed through a JavaScript application, and discuss sandbox tools.

The Network Layer

I’m going to start by briefly covering some computer science concepts. Full disclosure — my understanding of these ideas is basic. I don’t have a degree in computer science, and I’m not a network engineer. If you fall into one of those categories and have strong feelings about this post, please reach out! I would love to learn more.

Nonetheless, I will do my best to share the information I do know. I find these concepts extremely interesting. Without them, we wouldn’t have the vocabulary or architecture to make network requests in an application in the first place.

That being said, let’s talk OSI.

There is a model used in computer science called the “Open Systems Interconnection” (OSI) model. It dates back to the ’70s and is an academic, theoretical framework. It’s designed to be technology-agnostic so that we can continue applying the same principles to our networks as they develop over time.

OSI consists of seven layers, moving from the least abstract (the physical layer) to the most abstract (the application layer). I’ll walk through each of these one by one, but I wanted to give you a quick, colorful illustration. Feel free to visualize this as a cake. Or a parfait. Or a seven-layer salad. Whatever gets you through, I support it.

We start with the physical layer — the electricity connecting computers — and move all the way up to our application — an abstract series of requests and code that present our end user with something interactive.

Physical Layer

This is the electrical and physical representation of computer systems. When you think about the physical layer, think of radio frequencies and voltage protocols for things like Bluetooth and USB.

Data Link Layer

Also very low-level, the data-link layer is concerned with node-to-node data transfer using switches.

Network Layer

Getting more abstract, the network layer is concerned with large scale routing and connection. If a computer in San Francisco wants to communicate with a computer in Boston, the network layer will make it happen.

Transport Layer

The transport layer is where TCP/IP (Transmission Control Protocol) lives. HTTP relies on TCP/IP — it’s a protocol that standardizes requests. It outlines how we know that a connection has been made, how we break our request information into packets, and how we forward those packets, among other things.

Session Layer

The session layer handles the structure and behavior of a connection between two nodes (computers). How long does a computer wait for a connection to be made before it times out? If a request does time out, do we retry automatically? The session layer answers these questions.

Presentation

The presentation (or syntax) layer formats our requests. We might be requesting HTML, JavaScript or some raw data, but we need to agree on a langage for the requests themselves. Will they be encoded via JSON, YAML? Maybe ASCII? The presentation layer also handles encryption and decryption.

Application

The most abstract layer is the application layer — it’s also the layer that is most relevant to this discussion. This is where FTP, HTTP (Hyper-Text Transfer Protocol), Browsers and our application all live.

While most of what we talk about moving forward won’t reference OSI, it’s important to note that our application relies on each and every layer we just covered. When we’re trying to serve up some HTML over HTTP, it’s encoded, routed, and sent through a session via TCP/IP where switches and electrical impulses carry our info to the computer waiting at the other end. And it all happens in microseconds.

A gif of Bill & Ted’s Excellent Adventure with Ted (Keanu Reeves) saying “woah.”

Now that we have a framework in mind, let’s talk about how an HTTP or network request flows through an application.

You might notice that cache is not represented in this diagram. Caching is another very interesting and complex topic for another day. I am also specifically focusing on JavaScript and HTTP, so if you’re looking for info on Swift, Kotlin or SSL, this might not be the post for you.

It all starts with a user. When a user opens Firefox or Chrome or whatever the kids are using these days, and types in a URL (let’s say they want to play Spelling Bee), the browser routes their request. The application is waiting there, listening for that request, and responds by serving up any necessary code — HTML, JavaScript, CSS, static assets like images and logos, etc…

Oftentimes, in order to generate the correct code, the application will need to make its own requests (fetching puzzle data, for example). For that, our application will send a request to an API endpoint, which will then request info from a database.

Once we hit the end of the line, so to speak, the whole request chain reverses itself. The database will send data back to the API, which will send it back to our application, which will send it back to the browser, which will display it for the user. This entire loop happens within fractions of a second.

Debugging in the Browser

Let’s start by focusing on that first request — the one between the browser and our application. We can inspect and debug these requests using existing developer tools in your browser of choice.

Opening the network tab

Open developer tools from the browser menu (Firefox: Tools > Web Developer and Chrome: View > Developer) or by using cmd + option + i (Mac) and ctrl + shift + i (PC). Once you have your dev tools open, check out the network tab.

What we can debug in the browser

• Request format
• Request origin
• Headers
• Cookies
• Cache

Forgive my goofy screenshots, but these are a few of the features I find myself using the most in the network tab:

• Inspect request headers and cookies
• Searching for a request coming from a specific domain (look for the magnifying glass)
• Filtering requests by resource type (HTML, XML, JS, etc…)
• Manually adding and removing cookies (this lives in the storage tab, not the network tab, but I thought it was worth mentioning)

Use cases for debugging in the browser

I find the browser most useful for debugging network requests with a lot of architecture around them. This approach makes it more difficult to manipulate the request, but it’s quick and easy, and it also gets you very close to a production environment, which can be useful.

A few examples of where debugging in a browser might be useful:

• An outgoing tracking request that is formatted using complex headers along with data from your Redux store
• An incoming request that passes through multiple middleware functions where data is extracted and then passed down to the React app

Debugging in an Application

We can also take a look at the other side of the first request on our chart, and look at our application to see how it handles incoming requests with routing and middleware.

Routing will tell our requests where to go within our application, and middleware will examine a request and perform any necessary side effects.

Each app handles routing and middleware slightly differently, but there will generally be an entry point for your app’s back end (look for a server-side index.js file or something similar) using a framework like Express.

An example of an Express application with some middleware

What we can debug in an application

• Requests and responses
• How a request flows through our app
• Routing issues

Use cases for debugging in an application

The great thing about debugging in the app is that once you know where your application’s entry point and routing live, you can rely on your traditional debugging toolkit — unit tests, logging, error handling, etc…

A few examples of where debugging in an application might be useful:

• Issues with request headers or cookies not being processed correctly
• Issues where the request looks alright in the browser, but the application is not getting the request info as expected (this generally points to an issue with middleware)
• Routing issues like unexpected 404s
• Deep link errors where the incorrect URL is being generated

Debugging in a Sandbox

Now let’s take a look at the requests that our application sends out. Let’s say our application has received a request from the browser, but in order to respond with the correct page, it needs to fetch some information from the server.

Oftentimes, the application will make those requests through an API. Unless we’re actively developing this API and can spin it up locally, we have a little less insight here. We can only really inspect the format of our outgoing request and the response we receive. This is where a sandbox tool, good documentation, and collaboration with other developers is key.

Postman is one of my favorite request sandbox tools, but there are many available. They’re great because they allow us to make requests in isolation — it gives us complete control over what kind of headers, cookies and request data we would like to include. It’s the complete opposite of inspecting requests in the browser. If there is any architecture surrounding our requests in our sandbox, it’s because we put it there manually.

What we can debug in a sandbox tool

• Outgoing requests
• How cookies and headers behave in isolation
• Response data

Use cases for debugging in a sandbox

Setting up a request sandbox can take some time — you need to make sure you have the correct headers, cookies and authentication protocols in place — but once you have the framework, you can easily tweak your requests to test small changes in isolation. You can also save your requests and easily revisit them or apply the same request to multiple domains.

A few examples of where debugging in a sandbox might be useful:

• Mocking requests with different user or login profiles
• Mocking requests with small changes to cache keys, headers or query params
• Testing the same request quickly on local, staging and production environments
• Easily sharing complex request formats with teammates

Debugging Database Requests

We only have one more request on our chart — the one connecting an API to a Database. There is a whole world of options when it comes to getting information from your database. You could use one (or more!) of the following:

• ORMs (object relational mappers) like Sequelize
• Data layer/query languages like GraphQL
• Key-value look-ups in an object store like DynamoDB
• Raw SQL queries

Each of these approaches comes with its own suite of debugging tools, but the tools will often be familiar to the tools we discussed in this post. Apollo GraphQL, for example, has its own browser plugin, logging tools, sandboxing app.

A note on other connection types

Many of your application’s server-side requests will be happening over HTTP, but there are other secure ways to connect and communicate with a database. TLS, SSL and SSH, for example, all have their own use cases, but they’re outside of the scope of this post, so I will stop here for now!

Thank you for reading all of this! I hope you found it useful. In a gesture of gratitude, enjoy this cat photo:

Charlie loves to use my sewing box as a perch