knowledge

Overview

HTTP (HyperText Transfer Protocol) is the set of rules for communicating with web servers to transmit webpage data including HTML, images, videos, and other resources. HTTPS is the encrypted, secure version of HTTP using TLS/SSL. HTTP is a stateless protocol — each request is independent — and uses cookies to maintain session state across requests. All HTTP communications follow a client-server model: the client requests a resource and the server processes and returns it.

Terminology

TermDefinition
HTTPHyperText Transfer Protocol; rules for transmitting web data between clients and servers
HTTPSEncrypted version of HTTP using TLS/SSL
FQDNFully Qualified Domain Name — the complete domain name used to locate a host
URL (Uniform Resource Locator)Structured address specifying how to locate a resource on the internet
Stateless ProtocolEach request is independent; server retains no memory of previous requests
CookieSmall data stored on the client device to maintain session state with a server
Query StringAdditional parameters appended to a URL (e.g., ?id=1)
FragmentReference to a specific section within a resource (client-side only)
HeaderMetadata attached to HTTP requests and responses providing context and instructions
CRUDCreate, Read, Update, Delete — the four standard API operations
REST APIRepresentational State Transfer — architectural pattern for web APIs using HTTP methods
cURLClient URL — command-line tool for making HTTP requests and transferring data

Core Concepts

URL Structure

Resources over HTTP/HTTPS are accessed via a URL (Uniform Resource Locator). A Fully Qualified Domain Name (FQDN) is entered as a URL to reach a desired website.

ComponentDescriptionExample
SchemeProtocol to use; ends with ://http://, https://, ftp://
User InfoOptional credentials separated by @admin:password@
Host / DomainServer domain name or IP addressinlanefreight.com
PortSeparated from host with : — HTTP defaults to 80, HTTPS to 443:80
PathLocation of the resource; server returns index.html if omitted/dashboard.php
Query StringParameters passed to the server; multiple separated by &?login=true&user=admin
FragmentClient-side reference to a section within the resource#section-2

HTTP Flow

  1. Browser checks /etc/hosts first for local DNS overrides, then queries a DNS server to resolve the domain to an IP address
  2. A GET / request is sent to the resolved IP on the appropriate port
  3. The web server receives the request and processes it — returns index.html by default for root requests
  4. Server sends an HTTP response with a status code (e.g., 200 OK) and the file contents
  5. Browser renders the response and presents it to the user

/etc/hosts can be manually edited to add local DNS records — useful for redirecting traffic during testing.

HTTPS Flow

HTTPS encrypts all HTTP traffic using TLS/SSL. When a client connects with http:// instead of https://, the server issues a 301 Moved Permanently redirect to port 443.

Handshake sequence:

  1. Client Hello — client sends supported TLS versions, cipher suites, and random data
  2. Server Hello — server responds with chosen cipher and its SSL certificate
  3. Certificate Verification — client verifies the certificate and sends its own
  4. Encrypted Handshake — both sides confirm encryption is working; all subsequent traffic is encrypted

HTTP Downgrade Attack: An attacker can set up a MiTM proxy to intercept traffic and strip HTTPS back to HTTP, capturing plaintext data. Most modern browsers, servers, and HSTS headers protect against this. Even with HTTPS, DNS queries may reveal visited domains if not using encrypted DNS (e.g., 8.8.8.8, 1.1.1.1) or a VPN.

HTTP Methods

MethodPurpose
GETRetrieve a resource; parameters passed via URL query string
POSTSubmit data to the server; body carries parameters — used for forms, logins, file uploads
PUTCreate or replace a resource entirely on the server
PATCHPartially update an existing resource (vs. PUT which replaces entirely)
DELETERemove a resource — if unprotected, can be exploited for DoS
HEADFetch headers only; no body returned
OPTIONSList methods accepted by the server — useful for API enumeration
TRACEEchoes request back to sender; usually disabled
CONNECTEstablishes a secure tunnel (used for HTTPS proxying)

Most web apps rely on GET and POST. REST APIs additionally rely on PUT, PATCH, and DELETE. Use OPTIONS to discover which methods an API endpoint accepts.

HTTP Versions

VersionYearKey Features
HTTP/0.91991GET only
HTTP/1.01996Added headers and caching support
HTTP/1.11997Persistent connections; clear-text; fields separated by newline
HTTP/22015Binary data in dictionary form; improved performance and multiplexing
HTTP/32022Uses QUIC for faster, secure connections

HTTP Request Structure

Every HTTP request consists of four parts:

PartDescription
Start LineMethod, path, and HTTP version — e.g., GET /users/login.html HTTP/1.1
HeadersKey-value pairs providing metadata; terminated with a blank line
Empty LineDivides headers from body
BodyData payload — present in POST/PUT requests

HTTP Response Structure

Every HTTP response consists of:

PartDescription
Status LineHTTP version and response code — e.g., HTTP/1.1 200 OK
HeadersKey-value metadata
Empty LineDivides headers from body
BodyReturned content — HTML, JSON, images, PDFs, etc.

HTTP Status Codes

RangeCategoryCommon Examples
100–199Informational100 Continue
200–299Success200 OK, 201 Created
300–399Redirection301 Moved Permanently, 302 Found
400–499Client Errors400 Bad Request, 401 Unauthorized, 403 Forbidden, 404 Not Found, 405 Method Not Allowed, 408 Request Timeout
500–599Server Errors500 Internal Server Error, 502 Bad Gateway, 503 Service Unavailable, 504 Gateway Timeout

Servers and CDN providers (Cloudflare, AWS) may implement their own custom codes beyond this list.

HTTP Headers

Headers provide metadata about the request or response. Some are specific to requests or responses; others are general or entity headers common to both.

General Headers

Describe the message itself rather than its content.

HeaderDescription
DateDate and time the message originated; converted to UTC
ConnectionWhether the network connection should stay open after the request — keep-alive or close

Entity Headers

Describe the content being transferred. Found in responses and POST/PUT requests.

HeaderDescription
Content-TypeType of content being transferred; includes optional charset for encoding
Media-TypeSimilar to Content-Type; can influence how the server interprets input
BoundarySeparator marker when a message contains multiple content parts — e.g., boundary="b4e4fbd93540"
Content-LengthSize of the body in bytes; required for the server to read the message body
Content-EncodingCompression or transformation applied to the content

Request Headers

HeaderPurpose
HostTarget domain — important recon target; may indicate other virtual hosts
User-AgentIdentifies the client software and version
RefererURL of the page that originated this request — should not be trusted by servers
AcceptMedia types the client can handle; */* accepts all
Accept-EncodingSupported compression formats
CookieSession cookies sent to the server; multiple cookies separated by ;
AuthorizationAuthentication credentials — Basic <base64> or Bearer <JWT>
Content-LengthSize of the request body

Response Headers

HeaderPurpose
Set-CookieInstructs client to store a cookie; parsed by browser for future requests
Cache-ControlDefines caching behavior; no-cache prevents sensitive data from being cached
Content-TypeType of content returned
Content-EncodingCompression method used on the response
ServerReveals server software info — recon vector; version can expose known vulnerabilities
LocationRedirect target — can lead to open redirect if user-modifiable
WWW-AuthenticateInforms the client what type of authentication is required for the resource

Security Headers

HeaderPurpose
Content-Security-PolicyDefines allowed content sources; prevents XSS by restricting script/style origins
Strict-Transport-SecurityEnforces HTTPS; prevents protocol downgrade attacks
X-Content-Type-Options: nosniffPrevents MIME type sniffing attacks
Referrer-PolicyControls how much referrer info is sent during navigation

CSP example: Content-Security-Policy: default-src 'self'; script-src 'self' https://cdn.example.com HSTS example: Strict-Transport-Security: max-age=63072000; includeSubDomains; preload

Cookies

  • Small pieces of data stored on client devices to maintain session state across stateless HTTP requests
  • Set by server via Set-Cookie response header; sent back by client via Cookie request header
  • Used for session tracking, preferences, and authentication

Authentication

HTTP Basic Auth

HTTP Basic Auth is handled directly by the web server to protect specific pages or directories. Credentials are base64-encoded and sent in the Authorization header.

Authorization: Basic YWRtaW46cGFzc3dvcmQ= (base64 of admin:password)

Modern apps use JWT tokens instead: Authorization: Bearer <token>

Form-Based Auth (POST + Cookie)

Most login forms are built in server-side languages (PHP, etc.) and use HTTP POST to submit credentials. On success, the server returns a Set-Cookie header with a session token, which the browser stores and sends on subsequent requests.

Browser DevTools

Built into most modern browsers — not just for developers. Essential for web pentesting.

  • Open: F12 or Ctrl+Shift+I
  • Network tab — captures all HTTP requests made by the page
  • Filter URLs — search for specific requests
  • Click a request — view full headers, body, cookies, and response
  • Right-click → Copy → Copy as cURL — generates a full curl command replicating the exact request including all headers and cookies
  • Application → Storage → Cookies — view, edit, or manually add cookies with the + button

CRUD / REST APIs

APIs allow programmatic access to server resources. Many interact with a database where the table and row are specified in the URL.

API Standards

StandardDescriptionFormatBest For
SOAPSimple Object Access Protocol; rigid, XML-based messaging with a strict schema and stateful operation supportXMLEnterprise integrations, financial services, legacy systems
RESTRepresentational State Transfer; uses standard HTTP methods and URL paths; stateless and flexibleJSON, XML, or rawModern web APIs, public APIs, microservices

REST is the dominant standard for modern web APIs. SOAP is still used in legacy enterprise environments where strict data contracts and stateful transactions are required.

CRUD Operations

OperationHTTP MethodDescription
CreatePOSTAdd a new entry to a resource
ReadGETRetrieve entries; wildcard or partial matches often supported
UpdatePUTReplace an entire entry; PATCH updates a partial entry
DeleteDELETERemove an entry — unprotected endpoints are a vulnerability

PUT vs PATCH: PUT replaces the entire entry; PATCH applies partial modifications. Check OPTIONS header to confirm which the API accepts.

API interactions typically require authentication via Authorization: Bearer <JWT> or a session cookie. Use | jq to format JSON responses in the terminal.

Virtual Hosts

Allows a single web server to host multiple websites by mapping each domain to a different root directory.

  • one.com → /var/www/website_one
  • two.com → /var/www/website_two

Content Delivery Networks (CDN)

Distributes static website files across thousands of servers worldwide.

  • Routes requests to the nearest server rather than the origin
  • Reduces latency and improves load times for global users
  • HTTP Interception
  • Session Manipulation

References / Images