RFC 2616 is the definitive resource for HTTP/1.1. The IETF currently has a working group, Httpbis, who has a charter to revise and clarify RFC 2616. Their documents (currently drafts) will eventually replace RFC 2616. The drafts have gone through numerous revisions and I now look to them when looking for clarification.
I've learned to not trust HTTP implementations. Writing a proper HTTP stack is hard. Making it conform to all the corner cases is even harder.
When dealing with load balancers and proxies, I tend to avoid performing HTTP until absolutely necessary. In other words, put load balancers in TCP mode and don't have them talk HTTP. I trust a network appliance's ability to speak TCP more than HTTP.
One of the biggest mistakes I see people make with HTTP is improper usage of status codes. I attribute most of the mistakes to people reading the simple text description of the status code and not specification. This naive approach leads to completely wrong usage of codes like 406 Not Acceptable, which is used in relationship to the Accept- series of headers and has nothing to do with the general acceptability of a particular request.
HTTP provides a mechanism whereby you can perform conditional requests. Effectively the client says "I know X and Y about this resource: only retrieve if it is different." This is a very powerful construct. Proper understanding of conditional requests can mean the difference between effortless scalability and a costly hardware or software build-out.
Validators are means by which content/resources are compared. They come in two flavors: weak and strong.
Weak validators are typically cheaper to create and compare compared to strong validators. However, they aren't as effective.
Strong validators have the rule that they must change to a new, previously unused or guaranteed unique value whenever a change occurs to the representation data such that a change would be observable in the payload body of a 200 response to a GET. Two resources cannot share the same validator unless the paylad body would be identical. Strong validators should also be unique over all representations over all time.
An example of a weak validator is a time. You could take the time a file or record was last modified and use that as a validator. An example of a strong validator is the SHA-2 of a file.
Many people use hashes for strong validators. This is a good way to ensure the uniqueness requirements for strong validators, as the probability of hash collisions (assuming a proper algorithm) is extremely small.
The general rule is to use stong validators if you can. Strong guarantees are better than weak guarantees, after all. However, pay careful attention to how you are calculating the validators. Calculation could be expensive, possibly too expensive for your needs. You may be tempted to cache these strong validators. Be careful: can you guarantee the cache will always be updated if the underlying resource changes? If not, you will likely encounter bugs.
That being said, weak validators have their use. For example, you may want to force a conditional request based on resource age alone. Here, weak validators are perfectly fine.
It is common to use a date and time as a validator. These validators are typically weak, but they can be strong if some conditions are met. Read the spec for the crazy rules.
Dates work great as cheap validators under most circumstances. However, they don't work too well if:
The ETag header holds the validator for a particular representation. It is an opaque quoated string. If it is prefixed with W/, the validator is weak. Otherwise it is strong.
Unfortunately, there is no one-size-fits-all rule for validator generation. If you are serving files off a filesystem, the HTTP server will typically use mtime as a weak validator and MD5, SHA-1, etc of the file's contents as a strong validator. Unfortunately, the ETag header is sent at the top of the response, so the value must be calculated before the response body is sent on the wire. If the server were dynamically calculating file hashes, it would have to choose between reading a file once from I/O and buffering to memory or performing 1 read pass to calculate the hash and another to send the file over the wire. The real answer is to use a modern filesystem like ZFS where file hashes are stored as part of the file metadata and are updated automagically. However, most HTTP servers don't have these kinds of deep hooks to save work. In the end, you know your own application the best, so you know how to most effeciently implement validator generation.
Servers can return partial responses for resources.
Request has Range header. Could also have If-Range to make conditional. Response contains Content-Range OR Content-Type: multipart/byteranges with Content-Range for each part. Response must also contain Date and {Cache-Control, ETag, Expires, Content-Location, Last-Modified, Vary} if a non-range request would have resulted in a 200 and contained these.
Response code is 206 on success or 416 on failure.
Clients can request multiple ranges. There are all kinds of crazy rules around this. Read the spec.
It is possible for intermediary agents to transform responses. If they do, they are supposed to return 203 for originally 200 responses. If the original response wasn't 200, they should use the Warning header with the 214 code. I think they are supposed to retain the original response code at that point, but I'm not clear.
Representation transformation can be controlled by Cache-Control no-transform.
In the wild, many intermediary agents transform content without doing the right thing.
Not many stacks seem to have support for this. If writing an HTTP service, I would not rely on HTTP transformation unless you have total confidence in all HTTP stacks. In other words, only use this if you are a business deploying a private application on your network. If an open source project, you don't know what others' stacks will look like, so avoid relying on this.
Often used as part of look-before-you-leap conditional POSTs or PUTs dealing with large payloads. Can be in response to any HTTP method, however.
HTTP request must have Expect: 100-continue header.
Some stacks (like parts of .NET) utilize conditional POST or PUT automatically. Many (most?) HTTP server stacks intercept Expect: 100-continue requests before they hit the application layer. Apache is an example.
These responses can be sent after the server has seen part of the request. This may confuse poorly written HTTP stacks with strict state machines.
Some clients (can't remember which) send conditional requests but don't wait for this response.
If server sends 100 Continue, it must eventually send a final response.
Used if request contains Upgrade header. Part of switching the protocol of the TCP channel. Has nothing to do with application-level protocols.
Most common response code. It is almost always safe to send this. Fall back to this status code if nothing else makes sense.
Request resulted in new resource being created. Often used as response to PUT to a resource-centric URI.
Response's Location header contains URI for best representation of the created resource. This is typically the URI for the request.
Response body contains resource characteristics and location(s). Media type of response can be anything.
Resource must be created (and available) before server returns. If not, use 202.
Used for deferred processing. Server sends when it has received something but hasn't acted on it.
Response body should contain pointers so client can query processing state.
I've used this for job processing. Set up a bunch of workers processing jobs. Set up HTTP server. When client submits work, server assigns unique ID to job, saves it to a queue, then returns a 202 with metadata describing job and a URI to query for job state. Client then comes back later and queries job status URI to see if the job has finished. If not, the client typically waits and polls again. If it has, the client moves on, likely doing something with processed data.
For 200's that have been modified by an intermediary agent. See section about representation transformations for more.
Like a 200 but with no response body and special semantics. If server wishes to convey information about request, that is done in headers.
If you communicate any additional information in custom headers, I think it is best to use 200, 201, or 202 and send that as part of the response body instead.
Mostly used for interactive user agents. Server effectively tells client "I processed what you sent, reset your state and prepare to go again."
Contains no response body.
Most of the complexity here is in the client. I'm not sure what type of client support there is for this code.
I've never seen this used in the wild. I would avoid it unless it really, really makes sense.
Used in response to a partial content request. See the section on that.
The requested resource has multiple representations. Almost certainly sent in response to a GET.
Response body contains representation of different choices. You should contain URIs in this response.
If there is a best URI, that is sent in the Location header. Clients are allowed to auto redirect if Location header is set. Watch out for this trap! Not sure what popular agents do in practice.
When considering this status code, be sure you aren't reinventing content negotiation. If you don't want to rely on content negotiation and expose different media types at different URIs, this may come in handy.
Requested resource has been moved permanently.
Only use this if you are absolutely sure you'll never want to reuse the URI, as downstream agents may remember the permanent change and not ask the server again if things have changed. You can theoretically control this with caching headers, but don't rely on it.
New URI should be in Location header. It is valid to change the value of the Location header if things change later on. But, be sure that the original URI handed out by Location will also 3XX to new URI, or clients could get lost.
Resource temporarily at another URI.
This is an old HTTP 1.0 status code. Use one of the other 3XX codes if possible.
This code is often used incorrectly as a result of a POST. The semantics of this code are "that thing you requested is over here for the moment." if you are looking for "I processed your request and the results are here," use 303.
Agents are allowed to transparently convert from POST to GET when receiving this code. Consider 307 instead.
Provides an indirect response to a request.
This should be used when processing POSTs and you want to return a resource while maintaining URI purity. For example, if submitting form data with POST, the server processes that request without issue and wants to give the client a representation of the new state. The server would issue a 303 with Location pointing to the URI representing the result of the POST.
Sent in response to conditional requests and says the response has not been modified.
Deprecated in HTTP Bis drafts. Avoid this.
Been deprecated for a while. Don't use.
Similar to 302 except different semantics. If writing a new service and you don't need HTTP/1.0 compatbility, you can use this.
Agents may perform redirect automatically for safe methods, otherwise agent is required to confirm redirect. That seems to imply that this isn't suitable for automated agents??
The 4xx class is for client errors. They are frequently misused because people read the short name not the full description. Response bodies to 4xx should describe the error.
From the spec:
The server cannot or will not process the request, due to a client
error (e.g., malformed syntax).
There is much contention over whether this applies strictly to the HTTP protocol or whether it can be expanded to application-level as well. Many have started using this to indicate general failures with the request (e.g. missing a query string parameter).
In the ideal world, I think it would be OK to use this as a general purpose, application-level "I couldn't process the request because the client did something wrong, not me." In the real world, there are practical concerns. For example, some HTTP stacks will close the TCP connection when they see a 400 because they interpret 400 to mean the HTTP bits over the wire couldn't be parsed and the processing state is undefined. This is a completely valid concern! If you are an intermediary HTTP agent, you have to effectively guess who is right. Since you are allowed to close a persistent connection any time, many stacks play it safe and do this.
If you find your high volume web service behind an intermediary agent that closes connections after seeing a 400, you could be in a world of hurt. All of a sudden requests take a few ms longer to establish the socket. And then one day you get random 25s request times. You scratch your head for a few minutes then realize you have thousands of sockets in TIME_WAIT and you've exhausted your TCP socket pool because of many short-lived requests.
I think HTTP stacks have been trending towards not closing connections on 400. But, if you want to liberaly use 400, test on your infrastructure first and be sure to continuously monitor things by verifying new load balancer software updates, etc.
Unfortunately, HTTP doesn't have a general purpose "there is something generic wrong with the request" status code. So, you can take a (small?) risk with 400's, use a more appropriate 4xx, or send a 200 with a media type that reflects errors.
The request requires some form of authentication to be performed. Response contains WWW-Authenticate header. Client should then repeat with a suitable Authorization header.
Reserved for future use. This was originally conceived to be used for micropayments. Nothing official has manifested.
Some people have devised schemes utilizing this. Since nothing is standardized, I recommend avoiding it to avoid nonconformance with future specs.
Where 401 is authentication failed 403 can be thought of as authorization failed. The credentials and everything else in the request are valid, but they don't have access or aren't authorized to make the request.
Note that 403's can leak data by identifying whether a resource exists. For example, if user X attempts to access Y and receives a 403, they know that Y exists. If you are concerned about this leakage, you are allowed to return 404 even though 403 is more technically accurate.
Nothing found at the requested URI. Condition could be temporary or permanent. If permanent 410 is more appropriate.
This can also be used when the server doesn't want (or can't) reveal why a request is refused or when no other response is applicable.
Sent when the HTTP response method is not allowed for the current URI. Response contains Allow header containing comma-delimited list of methods supported. Most people forget the Allow header. It is required.
Requested content could not be delivered because the server wasn't able to satisy the client's parameters given in Accept and Accept- headers. In other words, if content negotiation yields nothing suitable, this is returned.
Many people read Not Acceptable and think it means a generic client-side error. This is wrong.
Response should contain details on what content is available so the client can do something intelligent instead of giving up.
See the section on content negotiation.
This is like a 401 but is used for proxy authentication.
The request was not produced/read in the time the server was willing to wait. Typically servers will send this automatically and applications running on the server have very little control over it.
This typically only comes up when dealing with slow clients or very large uploads.
The request could not be performed because it would result in a resource in a conflicted state. This is only allowed when the client might be able to resolve the conflict and try again. If the conflict can't be resolved, the spec says nothing on what should be used.
Since the client should be able to resolve the conflict, the response should contain details to aid this.
This is like a permanent 404.
Intelligent clients can remember 410's and may never try to request that URI again (because the condition is permanent). So, you shouldn't return a 410 unless you are absolutely positively sure that URI won't be used again. If you don't know, 404.
410's are useful to instruct others to remove references to them. For example, when a search engine spider sees a 410, it will typically deindex the page immediately, However, it may attempt to query a 404 again later.
The server requires a Content-Length header on the request.
This would likely only occur when performing chunked transfer and the server wants to ensure size restrictions before it starts processing the request.
Sent when a precondition (If-Match, If-None-Match, If-Modified-Since, If-Unmodified-Since, If-Range) evaluated to false. What's happening is client says "I'm asserting X and Y." Server says, "No, X isn't true," and sends a 412.
Known as Request Representation Too Large in HTTP Bis drafts.
The server refuses to process the request because it is too large.
This can be used to enforce size limits on posted entities.
The condition is allowed to be temporary and can be identifies as such by using Retry-After.
Known as URI Too Long in HTTP Bis drafts.
Sent when the URI (the bit in the first line of the HTTP request) is too long. HTTP servers typically do this automatically. Unless you are implementing an HTTP server, you probably have no justification for sending this.
While the HTTP spec doesn't define a length limit, most servers impose one. It is hard-coded in some servers and configurable in others.
Anything over 1024 bytes is probably pushing the envelope. If you see this error, you should probably be sending data via the HTTP request body, not the URI.
The server can't understand the Content-Type of the request.
Returned when performing a range request and none of the requested ranges could be fulfilled.
The client sent an Expect header and the server doesn't know how to handle it.
The server is requesting that the client change the protocol on the channel. The response contains an Upgrade header saying what to upgrade to. After that, the client is supposed to follow typical upgrade procedure.
This does not involve application-level upgrades. Instead, it is telling the client to change the underlying protocol away from HTTP/1.1 and to something else.
General server-side error code. If the server couldn't process the request because of an error on its end, this should be sent.
Many HTTP handlers will send this out in the case of an uncaught exception, other errors.
Sent when the server does not support handling a request.
If I'm implementing an HTTP service, I use this for URIs whose functionality I haven't coded yet.
An intermediary agent sends this when an error was encountered talking to an upstream server.
Denotes a temporary condition where the server or service is unavailable for whatever reason and can't or doesn't want to process a request.
Some servers send this automatically. Others will just not accept new sockets.
It is a good idea to send a Retry-After header to tell clients when they should try again. This can be used for backoff control. Of course, not all clients heed the advice and try again immediately, so you can't rely on it as a infallible failure mechanism.
An intermediary agent timed out while waiting for a response from an upstream server.
The protocol in the first line of the request message is not supported. This is typically sent automatically by the server.