A lightweight HTTP 1.1 web server implementation built in C++ for the 42 school curriculum. This server handles multiple connections via multiplexing (poll/select), serves static files, processes various HTTP methods, and supports CGI scripts.
Our team of 3 divided the project into key components:
- Configuration file parsing and server setup
- Request handling and response generation
- Socket management and CGI implementation
- Documentation, testing, and CI/CD pipeline
./webserv [configuration file]Our server implements both CGI support and file upload capabilities, following an nginx-like configuration syntax.
- Where it all started - The HTTP protocol and UNIX sockets
- RFC's
- The Project
- Multiplexing
- The tcp/ip sockets
- CGI
- Testing
- Some Definitions
- Resources
HTTP (Hypertext Transfer Protocol) was invented by Tim Berners-Lee at CERN (the European Organization for Nuclear Research) in 1989. The first version of HTTP, HTTP/0.9, was a simple protocol for transferring raw data across the internet. The more widely recognized version, HTTP/1.0, was specified in 1996, followed by HTTP/1.1 in 1997, which introduced persistent connections and other improvements. Today we have HTTP/2 and HTTP/3, but our server will need to handle HTTP/1.1 only.
RFC stands for "Request for Comments."
It is a type of publication from the engineering and standards organizations for the internet, such as the Internet Engineering Task Force (IETF) and the Internet Society (ISOC). RFCs are used to describe methods, behaviors, research, or innovations applicable to the working of the internet and internet-connected systems.
For this project, we will refer to the following RFCs:
- rfc2616: Hypertext Transfer Protocol -- HTTP/1.1
- rfc793: Transmission Control Protocol
- rfc791: Internet Protocol
- rfc3875: The Common Gateway Interface (CGI) Version 1.1
HTTP/1.1 was updated again in 2022:
- rfc9112: Hypertext Transfer Protocol -- HTTP/1.1
Because not everything is still relevant for us in the rfc2616, we summarized the most important parts. You can read a summary of the rfc2616 here rfc2616-summary.md.
And a summary of the rfc791 is here rfc793-summary.md.
The rfc3875 is not very long and we used it as reference for the CGI part of the project.
See the allowed_functions.md file here: allowed_functions.md
A server configuration file is a file used to define the settings and parameters for a server's operation. These files are essential for customizing the behavior of the server, specifying how it handles requests, manages resources, and interacts with other systems. Configuration files are typically written in a plain text format and can be edited using any text editor.
As per subject requirements in the configuration file, our server should be able to:
- Choose the port and host of each ’server’.
- Set up the server_names or not.
- The first server for a host:port will be the default for this host:port
- Set up default error pages
- Set the maximum allowed size for client request bodies
- Set up routes with one or multiple of the following rules/configurations: a list of accepted HTTP methods for the route, an HTTP redirect, a directory or file where the requested file should be located, enable or disable directory listing, and set up a CGI route.
a default file to serve when the request is for a directory. Allow the route to accept uploaded files and configure where they should be saved.
◦ Define a list of accepted HTTP methods for the route. ◦ Define an HTTP redirect. ◦ Define a directory or file where the requested file should be located - We can take inspiration from the ’server’ section of the NGINX configuration file
See here for more about configuration files.
For our project we decided to implement a syntax similar to the NGINX configuration file.
This is one of the key topic of this project for 42. As per subject requirements:
You must never do a read or a write operation without going through poll() or select() first. The core of the project is to handle multiplexing. It allows multiple connections to share the same network resources efficiently. In the context of web servers, multiplexing is used to handle multiple client connections without simultaneously without blocking or slowing down the server. We are not allowed to use multithreading except for the CGI server.
This is the main function simplified:
while (true) {
int status = poll(&pollfds[0], (nfds_t)pollfds.size(), POLLTIMEOUT);
if (status == -1) {
[...]
} else if (status == 0) {
[... timeout ...]; continue;
} else {
[ ... loop on the fds and handle the events ]
}
}where pollfds is a vector of pollfd structures. The poll() function will wait for events on multiple file descriptors. When an event occurs, poll() will return, and we can check which file descriptor triggered the event. We can then read or write data on that file descriptor as needed. If there are no new connections the poll() function will block to save resources and return 0 after the timeout. If there are new connections the poll() function will return and we can handle the events.
Sockets are a method of IPC that allow data to be exchanged between applications, either on the same host (computer) or on different hosts connected by a network. The first widespread implementation of the sockets API appeared with 4.2BSD in 1983, and this API has been ported to virtually every UNIX implementation, as well as most other operating systems. - Kerrisk, Michael. The Linux Programming Interface (p. 1136). No Starch Press. Kindle Edition.
During this project I read a lot from this book which I recommend.
From the configuration we have a list of ports and the server will create a tcp/ip socket for each port. We will then bind the socket to the port and start listening for incoming connections.
We will use the poll() function to monitor multiple file descriptors for events. The poll() function is part of the POSIX standard and is used to wait for events on multiple file descriptors. It is similar to the select() function.
The poll() function takes an array of pollfd structures, each of which represents a file descriptor to monitor and the events to watch for.
The pollfd structure has the following members:
- fd: The file descriptor to monitor.
- events: The events to watch for (e.g., POLLIN for data to read).
- revents: The events that occurred on the file descriptor (set by poll() after the call).
We create an array of pollfd structures to hold the server sockets. We are going to loop on those server sockets and set the events we want to monitor to POLLIN. We then call poll() to wait for events on the server sockets. When an event occurs, we check which socket triggered the event and accepting the connection will generate a new file descriptor, which we then add to the pollfd array. The client requests will arrive on this new file descriptor.
When a client sends a first request to the server, the server will receive the request on one of the server sockets, generate a new fd and add it to the pollfds. The server will then read the request from that file descriptor, parse it, and generate a response. The response will be sent back to the client using the same file descriptor. The important thing to not here is that we are going to read in a non blocking way. We will read a buffer size of bytes and then go back to poll() to check if there is more data to read. This way we will not block and give each connection a fair share of the server resources.
- We check if we got the headers. The delimiter for the headers is "\r\n\r\n". If we did not get the headers we will go back to poll() and wait for more data.
- Once we get the headers we check if it is a get request. in this case there is no body so we can pass the headers to the response generator.
- if it is a multipart request we will read until we have the body and pass the headers and the body to the response generator.
- if it is a chunked request, since we do not handle them we will return a 501 Not Implemented response.
- in the CGI case, we need to create a child process, pass the headers in the environment variables, and pass the body to to the stdin of the child process. There will be two pipes at the moment of the creation of the child, one for the stdin and one for the stdout. We will add the pipe stdout and the pipe stdin to the pollfds.
Adding the pipe stdin to the fdpoll will allow us to return to poll and read from client and write to the child until end of body. After that the child might send us some data and we will write it to the client passing through poll again until the child process is done.
CGI stands for Common Gateway Interface. It is a standard protocol used to enable web servers to execute external programs, typically scripts, and generate dynamic content for web pages. CGI scripts can be written in various programming languages, including Perl, Python, and C/C++.
Key Points about CGI:
- Dynamic Content: CGI allows web servers to generate dynamic content based on user input or other data.
- Language Agnostic: CGI scripts can be written in any programming language that can read from standard input and write to standard output.
- Execution: When a web server receives a request for a CGI script, it executes the script and sends the output back to the client as an HTTP response.
- Environment Variables: CGI scripts receive information about the request through environment variables, such as QUERY_STRING, REQUEST_METHOD, and CONTENT_TYPE.
For example:
A user submits a form on a web page.
The web server receives the form data and passes it to a CGI script.
The CGI script processes the data (e.g., querying a database).
The script generates an HTML response based on the processed data.
The web server sends the HTML response back to the user's browser.
Here is a simple example of a CGI script written in python that outputs "Hello, World!":
#!/usr/bin/python
print("Content-Type: text/html")
print("content-length: 48")
print()
print("<html><body>")
print("<h1>Hello, World!</h1>")
print("</body></html>")To use this script, you would place it in the CGI directory of your web server (often cgi-bin), and configure the server to execute it when accessed via a specific URL.
Interestingly the script above has 3 details that often are overlooked.
1 - The first line is a shebang line that tells the operating system which interpreter to use to run the script. So the script is an executable.
2 - The Content-Type header is set to text/html, which tells the browser how to interpret the response. But more importantly, the first header is missing, the HTTP version and the status code. The server will add it automatically. But how does the server know which status code to add? The server will add a 200 OK status code if the script exits with a 0 status code. If the script exits with a non-zero status code, the server will add a 500 Internal Server Error status code.
3 - The script must output a blank line after the headers to indicate the end of the headers and the beginning of the response body.
If the shebang path is not correct, the server will return a 500 error. If the content length is not present the server might try to chunk the response (better) or return Constants::BUFFER_SIZE and truncate it (less ideal but depends of the project requirements).
See the testing.md file here: testing.md and also how we use siege here siege.md
- Socket: A socket is an endpoint for communication between two machines over a network. It can be used to send and receive data, establish connections, and perform other network-related tasks. Sockets are identified by an IP address and a port number and are in the OSI layer 4 (Transport Layer) of the network stack.
- Port: A port is a communication endpoint in an operating system that allows multiple processes to use the same network interface. Ports are identified by numbers ranging from 0 to 65535 and are used to direct network traffic to specific applications or services running on a machine and are in the OSI layer 4 (Transport Layer) of the network stack.
- TCP/IP TCP/IP (Transmission Control Protocol/Internet Protocol) is a suite of communication protocols used to connect devices over the internet. It provides reliable, end-to-end communication between devices by breaking data into packets and routing them across networks. TCP/IP includes protocols like TCP, IP, UDP, and ICMP. more about the TCP handshake here: tcp-handshake.md
More definitions are here: definitions.md
- HTTP/1.1 Standard RFC2616 - The Hypertext Transfer Protocol (HTTP) specification used by the World Wide Web since 1990
- TCP Protocol RFC793 - Transmission Control Protocol specification
- IP Protocol RFC791 - Internet Protocol specification
- Mozilla Developer Network - HTTP Overview - Comprehensive HTTP documentation
- Beej's Guide to Network Programming - Classic guide by Brian "Beej" Hall
- C++ Standard Library Reference - Comprehensive reference for C++ standard library
- Sockets and Network Programming in C - Blog tutorials on socket programming
- Files and Sockets Tutorial - Video guide on handling files and sockets
- NGINX Web Server - Source code of the popular NGINX web server
- Caddy Web Server - A modern web server with automatic HTTPS
- CGI on NGINX - Stack Overflow guide
- Server World CGI Setup Guide - Tutorial for CGI setup
- File Upload with CGI - Guide for handling file uploads with CGI
- ZeroSSL - Free SSL certificate issuance
- Don't Reinvent the Wheel - Blog post on code reuse
- GitHub Pages with Markdown - Setup guide for GitHub Pages documentation
- HTTP Protocol Comparison - HTTP/1.1 vs HTTP/2 vs HTTP/3
- Linus on Git - "Linus did it in 5 days, see where you can git"
- 404 Page Example - Example of a well-designed 404 page
- Doxygen Graph Examples - Helpful for documentation
- Git Fixum Guide - How to fix Git mistakes
- Dang It, Git! - Solutions for common Git problems
