httpc

An HTTP/1.1 client — the mirror of http, which serves. Speaks http:// and https://, follows redirects, and decodes chunked responses.

import "httpc";
import "time";

let dl = until_of(time.mono() + 5000000000);   // 5s for the whole request

let r = httpc.get("https://api.example.com/users?id=1", dl);
guard let resp = r else let e = err_of(r) {
    log.error("request failed: " + e);
    return;
}
println(to_str(resp.status) + " " + to_str(len(resp.body)) + " bytes");

// a POST, and a header the request owns
let req = httpc.new_request("POST", "https://api.example.com/users");
req.headers["Authorization"] = "Bearer " + token;
req.body = to_bytes(payload);
let r2 = httpc.send(req, dl);

Request carries method, url, headers, body, max_redirects (default 5, 0 disables following) and ca_path ("" = the system trust store). Response carries status, status_text, headers, body and url — the last being the URL that actually answered, which after a redirect is not the one you asked for.

The deadline covers the whole request, connecting included: a host that never answers the SYN fails at the deadline, over plain TCP and TLS alike.

Separate from http rather than folded into it because http imports only byteutil, while a client necessarily imports net — and for a TLS request that drags -lssl/-lcrypto onto the link line of every program that merely wanted to serve HTTP.

Five things worth knowing:

• A 404 is a Response, not an err. The result is about whether the exchange happened — DNS, connect, TLS, framing. A server that answers "no" answered. Collapsing the two would make a 404 and a connection refusal indistinguishable at the call site, and they need different handling.
• Certificates are verified, and there is no flag to stop that. Verified against expired, self-signed and wrong.host on badssl.com: all three are refused, a valid one is accepted. For an internal service signed by a private CA, set ca_path to that bundle — the answer is a different trust anchor, never a disabled check.
• Credentials do not survive a cross-origin redirect. Authorization, Cookie and Proxy-Authorization are dropped when the scheme, host or port changes, because the server that sent the Location chose where it points, and that is exactly how a token gets exfiltrated. Same-origin redirects keep them.
• Redirect method rules follow browsers, not the RFC's original wording. 303 always becomes GET; 301 and 302 after a POST also become GET and drop the body, which is what every browser and curl do and therefore what servers expect. 307 and 308 exist to preserve the method, so they do. A redirect loop stops at max_redirects and hands back the last 3xx rather than spinning.
• Everything a server can make you allocate has a ceiling — 64 KiB of headers, 32 MiB of body, and a bounded chunk-size line. A client talks to servers it does not control, so a buffer sized on their say-so is a denial of service arriving through an ordinary call.

Connection pooling#

httpc.get and friends are one-shot: a connection per request, closed afterwards, with Connection: close sent so the server does not hold it open. A Client keeps idle connections and reuses them — up to max_idle_per_host per origin (default 4), for idle_timeout nanoseconds (default 30s), and 64 across all origins.

let c = httpc.new_client();          // share one across tasks
let r = httpc.client_get(c, "https://api.example.com/v1/items", dl);

The one-shot functions ARE a client — one that keeps nothing — so framing, redirects, decompression and every security rule are one code path, and cannot drift between the two.

A Client is safe to share between tasks: the pool is behind a mutex. Its dials and reuses fields and httpc.idle_count(c) make its behaviour checkable rather than asserted.

• Every pooled connection is probed before use. Servers close idle connections on their own timers (Node's default is 5 seconds), and a request written onto one fails indistinguishably from the server failing mid-request. The probe is net.idle_alive — one non-blocking MSG_PEEK, no latency, nothing consumed.
• A dropped request is retried once, and only if it is idempotent. If a reused connection dies before a single response byte arrives, GET/HEAD/PUT/DELETE/OPTIONS/TRACE retry on a fresh connection. POST does not: it may already have been acted on, and sending it twice could charge a card twice. The probe is what protects a POST; the retry is the backstop for the race between probe and write.
• The pool key includes ca_path. A connection verified against one trust anchor is never handed to a request that asked for another — tested over real TLS: the second request, demanding a different CA, fails verification instead of riding the verified connection.
• Reuse requires a clean end: HTTP/1.1, no Connection: close, a framed body read exactly, and nothing left over. A body delimited by the connection closing is never reused.

Client operations come in two equivalent forms: handle-first package functions — httpc.client_get(c, url, dl), the same idiom as sql.exec(db, ...) — and methods, c.get(url, dl). The methods are get, head, post, send, idle_count, close_idle, enable_cookies, clear_cookies, set_cookie and cookies on Client, and header on Response. c.get and the one-shot httpc.get share a name; the method is the pooled client's, the function is not.

Decompression#

Requests carry Accept-Encoding: gzip, deflate and responses are decoded transparently — unless the caller set Accept-Encoding themselves, in which case the body comes back exactly as sent. A caller who asked for gzip wants the gzip (to proxy it, to store it), and decompressing behind their back would hand them something else.

• The decoded size is held to the same 32 MiB ceiling as a plain body. Without that the wire limit would mean nothing: 32 MiB of gzip holds tens of gigabytes.
• deflate is tried as zlib (what RFC 9110 says it means) and then as raw DEFLATE (what a real share of servers send under that name).
• After decoding, Content-Encoding and Content-Length are removed: both describe the bytes that crossed the wire, not the body in hand.
• An encoding the client did not ask for and cannot read is left alone, header included — visible, not silent.

Cookies#

RFC 6265, with RFC 6265bis's rules for Secure and the __Secure- / __Host- prefixes. Off by default — the opposite of a browser, on purpose:

let c = httpc.new_client();
httpc.enable_cookies(c);             // this Client now acts as ONE identity

A browser's jar belongs to one person. A server's Client is usually shared, one per process, used on behalf of every user it serves, and a jar there would store user A's session cookie and send it on user B's request. Go's http.Client makes the same call. Turn the jar on for a Client that represents one identity: a scraper, a test driver, an integration with a login-based API.

• Cookies are applied per redirect hop. A cookie set by a 302 reaches the page it points to, which is how nearly every login works; a redirect to another host carries that host's cookies, not the first one's.
• Refused: a Secure cookie from http:// (a network attacker could otherwise overwrite an https session); a Domain the host does not belong to; a bare single-label Domain such as com; a Domain that is part of an IP address; __Host- without Secure, or with a Domain, or with a path other than /; a line with no =; anything over 4096 bytes. Secure cookies are only ever sent over https.
• Scoping: a cookie set without Domain goes back to exactly that host (127.0.0.1 and localhost are different scopes); Path=/p matches /p and /p/x but not /pathx. Longer paths are sent first.
• Expiry: Max-Age wins over Expires whatever order they arrive in, and a past date or Max-Age=0 deletes. Dates are parsed with RFC 6265's lenient algorithm, since servers send RFC 1123, RFC 850 with two-digit years and asctime alike; httpc.parse_cookie_date exposes it.
• Limits: 50 cookies per domain and 3000 in total, evicting the oldest, so a server flooding the jar pushes out its own cookies first.
• Response.set_cookies holds each Set-Cookie line separately. Use it rather than headers["set-cookie"]: repeated headers are joined with ", ", and an Expires date contains a comma, so two joined lines cannot be split apart again.

No public-suffix list is applied, and the consequence is stated rather than left to be discovered: a response from a.example.co.uk may set a cookie for Domain=co.uk, and this jar will then send it to every *.co.uk host. Nothing in slang ships a suffix list, and an embedded copy goes stale. The bare-TLD rule stops Domain=com; it does not stop that case.

Not done, deliberately: no HTTP/2 client, no multipart bodies, no proxy support, no public-suffix list.

API#