[6740] in www-talk@info.cern.ch
HTTP-NG: status report
daemon@ATHENA.MIT.EDU (Simon E Spero)
Sun Nov 20 13:01:10 1994
Date: Sun, 20 Nov 1994 18:46:53 +0100
Errors-To: listmaster@www0.cern.ch
Reply-To: ses@tipper.oit.unc.edu
From: Simon E Spero <ses@tipper.oit.unc.edu>
To: Multiple recipients of list <www-talk@www0.cern.ch>
------- =_aaaaaaaaaa0
Content-Type: text/plain; charset="us-ascii"
Captain's log, stardate 11-20-94.
We have been in orbit of Bristola system for 2 weeks. Acting under orders
from W3 headquarters, an away team from engineering has been working on
the planets surface attempting to renovate the native hypertext system.
Their report now follows.
---
Here's some quick notes on how HTTP-NG is coming along - We've now got
a simple proof-of-concept implementation up and running, using a
minimalist set of requests and responses, and the results have been
quite encouraging. Even though the network connection between Bristol
and Civillization was extremely congested, initial timings seemed to
show that HTTP-NG was capable of using the entire path bandwidth
available when multiple requests were used. Initial measurements
seemed to show that for typical inlined-objects, HTTP 1.0 would only
use around a 10th of the available bandwidth, and that parallel
fetches degraded badly under congested conditions.
Architecture.
HTTP-NG is designed to be a replacement for HTTP 1.0 offering much
higher performance and adding some extra features needed for use in
commercial applications. The protocol is designed to make it easy to
implement the basic functionality needed for a simple browser whilst
making the addition of more powerful features such as security and
authentication much simpler than it is for HTTP.
HTTP-NG uses a different basic model to HTTP. HTTP sets up a new
connection for every request, which causes a lot of severe performance
problems both in the time taken for each transaction, and the load
placed on both networks and servers.
To avoid these problems, HTTP-NG allows many different requests to be
sent over a single connection. These requests are asynchronous -
there's no need for the client to wait for a response before sending
out a different request. The server can also respond to requests in
any order it sees fit - it can even interweave the data from multiple
objects, allowing several images to be transferred in "parallel".
To make these multiple data streams easy to work with, HTTP-NG sends
all it's messages and data using a "session layer". This divides the
connection up into lots of different channels. HTTP-NG sends all
control messages (GET requests, meta-information etc) over a control
channel. Each object is returned over in its own channel.
This separation of data and control information also makes redirection
much more powerful - for example, if the object is a video the server
can return the meta-information over the same connection, together
with a URL pointing to a dedicated video transfer protocol that will
fetch the data for the relevant object. This becomes much more
important when working with multimedia aware networking technologies,
such as ATM or RSVP.
----
Implementation experience.
There are several obvious ways of implementing HTTP-NG. The first
approach is to use a simple synchronous RPC model. The client
establishes a connection to the server, then sends out a request. The
client then waits until it has received all the data corresponding to
the request, then returns. This approach is very simple to implement,
and corresponds quite closely to current practice in older HTTP
clients. As a simple optimization, the implementation could send a
number of requests before waiting for all their responses.
The second approach is to use an event based model. A simple
dispatcher can be wrapped around the connection. Each request then
adds a callback that should be called whenever the response to the
request comes in. This model works well with event based browsers such
as Arena and Netscape; it is also quite simple to implement.
The final approach is to use a multi-threaded implementation; each
request and channel gets allocated its own thread, allowing multiple
requests to be processed in parallel. This model works supremely well
for servers, especially those serving as proxies. The proxy server
can keep a cache of open connections to other proxies or to popular
sites. Since each connection resides within a single process, requests
that can't be satisfied from the local cache can easily be switched to
the appropriate connection. This approach also allows efficient
gatewaying of HTTP 1.0 requests (see next section).
Transition
----------
The best transition strategy for moving from HTTP 1.0 to HTTP-NG is
through the use of intermediate proxy servers. This allows the
existing base of servers and clients to continues operating as they
are now, whilst still taking advantage of much of the performance
enhancements in the new protocol.
The reason that this works is that most of the performance problems in
HTTP 1.0 are caused by delays in the network. If proxy servers are
placed close to older clients and servers, then these delays become
significant. For example, if two servers are placed at either end of a
transatlantic link, communicating with each other using HTTP-NG, but
accepting and sending requests to and from other systems using HTTP
1.0, all the HTTP 1.0 delays would all occur within a continent,
rather than spanning the intercontinental links. Further, a cacheing
server can interpret HTML documents and pre-fetch any inlined objects
before an HTTP client requests them.
------- =_aaaaaaaaaa0
Content-Type: text/plain; charset="us-ascii"
Content-Description: Architectural overview (text dump from framemaker)
FHTTP - Architectural Overview
Simon Spero, UNC Sunsite/EIT
(ses@unc.edu)
About FHTTP
FHTTP is an enhanced replacement for HTTP/1.0. It is
designed to correct the known performance problems in previous
versions of HTTP, and to provide extra support for commercial
transactions, including enhanced security and support for on-line
payment.
About this document
This document provides an architectural overview of the new
protocol. It gives an overall view of how the protocol works, and
explains how various operations and features interact with each
other and the network. This document is intended for a general
audience, and is not a technical specification. This is not an
official W3O document.
About this series
This document is part of a series describing FHTTP.
1. Architectural Overview
2. The Session Control Protocol
3. Introduction to ASN.1 and the Packed Encoding Rules
4. FHTTP Specification
5. Security Profiles
6. Payment Profiles
Why do we need a new protocol?
HTTP is the fastest growing protocol on the internet. It is simple to
implement, and thousands of people use it every day to browse through
gigabytes of on-line hypertext. If HTTP is working so well, why
replace it? The driving forces behind this change are the twin needs
of performance and commerce.
The Need For Performance
Part of the need for a new protocol is caused by the very fact of
HTTP's success. The original protocol was designed to be a simple way
of transferring a file between two machines. This led to several very
serious performance problems, adding extra delays to the time taken to
fetch pages, and preventing browsers from making efficient use of the
network.
Commercial Applications
Although many publishers and companies are now starting to explore the
possibilities opened up by the World Wide Web, the protocols currently
in use do not support several very important features needed for
electronic commerce. Currently there is no reliable way to find out
the identity of someone trying to access a document. There is also no
way to convey information about charging and prices; nor is there a
way of dealing with on-line payment.
Can these changes be made by just tweaking HTTP 1.0?
The major problem in making just minor changes HTTP 1.0 is that the
modifications which are needed change the fundamental model on which
the protocol is based. Each extension adds more and more com- plexity
to the protocol; as more and more enhancements are made, the problem
becomes much worse. Since HTTP 1.0 is a simple protocol, it is much
simpler to recreate the existing functionality as part of a new
protocol than to attempt to kludge the new functionality into a
protocol not designed to handle such changes.
User Requirements
Any new protocol needs to meet the needs to three different groups.
Individual users want to be able to browse the web without being
forced to wait for pages to be delivered. Information providers need
to be able to support large numbers of users, and to restrict access
to authorized users, and to get money from their paying customers.
Software developers need a system which is easy to implement, but
which can be optimized and enhanced to differentiate between products.
TABLE 1. User requirements
Simplicity
FHTTP must allow simple implementations to be implemented simply
without penalising more optimized systems. The protocol should be
designed to work well for the commonest cases
Performance:
FHTTP should allow objects to be transferred over wide
area networks efficiently.
Asynchronicity:
FHTTP should allow a client or server to initiate a new
request with- out waiting for previous requests to complete. It should
be possible to transfer multiple objects in parallel over a single
connection.
Security:
FHTTP should support the transfer of encrypted
objects. The protocol should not impose a single security policy or
mechanism.
Authentication:
FHTTP should support mutual authentication between all parties
involved in a transaction. It should be possible to relay
authentication information through multiple untrusted
intermediaries. It should also be possible to use multiple
authentication contexts over a single con- nection.
Charging:
FHTTP should provide support for on-line payment schemes such
as First Virtual and DigiCash. The protocol should not impose a
single payment policy or mechanism.
Intermediate Servers:
FHTTP should provide support for server to server interaction to
allow for the maintenance of caches and mirrors, and to allow
intermediate servers to relay usage information to originating
sites.
Mandatory display:
FHTTP should support the mandatory display of information
relating to an object, such as licensing information, copyright,
authorship.
Logging Information
FHTTP should support the transfer of logging information between
intermediate servers and the original source of a resource. Users
should be able to specify restrictions on the use of logging
information, and to discontinue transactions if such restrictions
cannot be satisfied.
Network requirements - Transports of Delight
FHTTP is transport-layer independent, and can arrange for data
transfer using different transports. However, since all uses of FHTTP
in the current internet will be over TCP, FHTTP must work well over
TCP.
TCP and the Internet
Many performance problems in HTTP are a result of not taking Transport
layer interactions into account. Table\x112 lists some important
points that need to be taken into account when designing a proto- col.
TABLE 2. Important features of TCP
Connection setup costs
When TCP sets up a connection, it sends connection request to the
server, and waits for the connection to be accepted or rejected. This
adds a delay of one Round Trip Time.
Slow start limits transfer rates during start up.
When a connection is first started, TCP initially sends only a small
amount of data. The amount of data that can be transmit- ted before
the sender must wait for a reply is increased until a steady state is
reached- if congestion occurs and packets are lost, the server slows
down the send rate until a new stable point is reached.
Slow-start particularly affects the first request on a new connec-
tion. If the request won't fit into a single segment, the client must
wait an extra Round Trip before it can finish sending the request.
Congestion information is not shared between connections
Although slow-start is used to converge upon the correct trans-
mission rate for the path between two computers, this informa- tion is
not shared between different connections to the same host. Thus is
there are several connections running between a pair of hosts, if the
path becomes congested, the connections will interfere with each
other, leading to poor throughput.
Special purpose transport layers.
As resource reservation and guaranteed bandwidth become more
commonplace, and as ATM moves into more general deployment, it becomes
important to be able to make use of special purpose transport layers
which are optimised for certain types of media- for example, a
multicast video transmission, or a ATM telephone channel. FHTTP should
be able to refer a user on to another service for actual data trans-
fer, whilst still being able to handle the relevant control
information.
Protocol Model
HTTP 1.0 works by creating a new transport connection for each
request. The client sends a request over the connection; the server
replies over the same connection, sending information about the
response, followed if possible by the requested data.
FHTTP uses a different model. Instead of creating a separate
connection for each request, FHTTP cre- ates a single connection
which can be used for many different requests. The connection is
divided up into a number of virtual sessions. One of these
sessions is used to carry control information - requests from the
client, and meta-data from the server. The other channels are used
to carry the requested objects.
Requests and Responses
Each request and response is sent as an FHTTP message. Some parts of a
message are the same in all cases. For example, each message may need
to be signed by the sender to make sure that the request is genuine.
Other parts of the message are different for each operation. A GET
request will specify a list of the objects to fetch- the response will
contain information describing the retrieved objects.
All of this information must be encoded in some way before it can be
sent over the network. HTTP 1.0 uses a text based syntax to encode
requests, and a modified form of MIME to handle responses. This tex-
tural representation is easy for humans to understand, but quickly
becomes extremely complicated when used to handle requests with
complicated structure.
To avoid this complexity, FHTTP uses a different way of describing and
encoding the request message. The scheme used is a simplified form of
ASN.1 and PER (Abstract Syntax Notation, and Packed Encod- ing Rules).
This scheme allows efficient, compact parsers to be generated
automatically, whilst remain- ing simple enough to allow hand-crafted
parsers to be built easily.
FHTTP messages can be sent at any time. In the typical case, the
client sends a request message to the server, and gets back one or
more responses in return. The client doesn't have to wait for a
response to one request before sending out another. Since returned
objects can be interleaved on a stream, this allows clients such as
Mosaic Communications NetScapeTM to request objects as soon as they
discover they are needed, and allows intermediate servers to handle
requests from different clients to the same server in a fair and
efficient manner.
Although the usual pattern is for the client to issue requests to the
server, there are some cases where this pattern needs to be reversed.
One example for this is the need to warn browsers if a requested
action would result in a hefty charge. In this case, the server needs
to be able to check to see if the client still wishes to perform that
action, and to discuss with the client how sir wishes to pay.
Negotiations - The Art of The Deal
Because FHTTP doesn't restrict the types of objects which can be
requested, and does not impose a single security or payment policy,
there needs to be some way of negotiating which types and mecha- nisms
a client and server can support.
HTTP 1.0 allowed clients to propose a list of types which they were
able to accept; unfortunately the mechanism used had several problems.
Because the client had to sent a list of all possible types with each
request, requests typically contained over 1K of type information.
This caused serious performance problem, and required a lot of effort
to process correctly. As a result of this complexity, very few servers
properly supported this type negotiation.
FHTTP tries to simplify negotiation by adding an extra mechanism to
support the most common cases without affecting more complicated
situations. This new form of negotiation is based on the observation
that the vast majority of all WWW traffic involves the exchange of
just a few well-known object types. FHTTP defines a small list of
these well-known types, and allows sets of these types to be encoded
in a short bitmap. A text only browser would sent a bitmap indicating
support for just HTML and plain text. A graphical brower would claim
support for several graphics and sound formats. A server or browser
which only supports these types need only support this simple form of
negotiation.
To indicate support for other types. FHTTP also allows the client and
server to add extra items to the bit-sets. The proposer send a message
containing the option being proposed (for example, text/ms- word),
together with a numeric code which will be used in later messages to
refer to the proposed value.
This kind of indirect reference is useful, because it allows the
proposed types and values to become more structured without affecting
performance. This extra freedom can be used to support parameterized
types; for example, if a client is running on a 4-bit display, a
server can avoid generating a 24-bit deep image.
The same method is used to indicate supported security schemes,
authentication information, and pay- ment mechanisms.
Security
Because there are so many different security schemes and policies,
FHTTP provides a general security framework into which the various
security components can be fitted. This results in several
architectural decisions. One such is the absence of any special
support for certificate exchange; this exchange is han- dled using the
Fetch request (which is after all the most fundamental operation in
FHTTP).
The FHTTP message wrapper has fields which can carry arbitrary
authentication and security informa- tion. This allows each message to
be individually authenticated.
FHTTP also allows an intermediate server to relay authorisation and
verification to and from another server on behalf of one of its
clients. This feature allows untrusted proxies to cache encrypted
data, and to relay the information needed to decrypt this data to the
end user without having to be able to decode the document itself.
Charging and Payment
FHTTP allows a server to initiate a payment request in response to an
action of the client. This allows the server to request specific
authorisation from the client for actions for which charges will be
incurred- the challenge includes cost information, and a list of
acceptable payment mechanisms; the response may include payment using
a mutually acceptable method of on-line payment.
The Client may also use the negotiation mechanisms to authorise a
series of payments without requir- ing explicit approval for each
item. The server can also provide pricing information as part of an
objects meta-data.
Mandatory Display
The client may negotiate a set of fields such as licensing information
and authorisation which it is pre- pared to automatically display to
the user along with the actual requested information. This allows
infor- mation providers whose database licences require such
information to be displayed at the start of each session to be made
available over the web.
Copyright Control
Each transferred object may have associated copying information
associated with it. This information can be used to assist honest
implementations from accidentally violating the terms of a licence
agree- ment.
Transition from HTTP 1.0
FHTTP uses two strategies for transition from HTTP 1.0 to the new
protocol. One strategy uses a dual stack approach - the other is based
on the use of proxy servers.
To support the dual stack approach, FHTTP has been designed to
co-exist with FHTTP on the same port. Each FHTTP connect begins with a
syntactically legal HTTP 1.0 request, using a non-existent method. If
an FHTTP client tries to talk to an old HTTP server, the HTTP server
will send back a `Method not supported' message; the client can then
cache this information, and repeat the request using the old protocol.
If an old server tries to access a new server, the server can either
handle the transaction using HTTP 1.0, or return an error message.
The dual stack approach doesn't offer much improvement for clients
that don't support FHTTP directly. Proxy servers allow old clients to
take advantage of some of the newer performance features. The proxy
server can pre-fetch inlined images using the new protocol, so that
they are ready and waiting when the client requests them; since the
proxy server will usually be on the same network as the client, this
reduces latency delays to a minimum.
A proxy server can also multiplex several HTTP 1.0 requests over the
same connection. This allows browsers like NetScape to request
multiple concurrent images without causing congestion problems.
Historical Notes
This document was produced with assistance of David St. John Raggett
(Hewlett Packard), Phillip Hallam-Baker (CERN), Allan Schiffman (EIT),
and Tim Berners-Lee (MIT).
FHTTP has been inspired and influenced by several earlier protocols. The roll of honor includes:
HTTP (HyperText Transfer Protocol)
WAIS (Wide Area Information Servers)
NISO Z39.50 (Search and Retrieve)
IMAP (Internet Mail Access Protocol)
TCP (Transport Control Protocol)
T/TCP (Transaction TCP)
Bibliography
HTTP 1.0
HTTP-PROB
------- =_aaaaaaaaaa0
Content-Type: text/plain; charset="us-ascii"
Content-Description: Session control protocol
Session Control Protocol (SCP)
------------------------------
Several heavily used Internet applications such as FTP, GOPHER, and
HTTP use a protocol model in which every transaction requires a separate
TCP connection. Since clients normally issue multiple requests to the
same server, this model is quite inefficient, as it incurs all the connection
start up costs for every single request.
SCP is a simple protocol which lets a server and client have multiple
conversations over a single TCP connection. The protocol is designed to be
simple to implement, and is modelled after TCP.
Services.
---------
SCP's main service is dialogue control. This service allows either end of the
connection to establish a virtual session over a single transport connection.
SCP also allows a sender to indicate message boundaries, and allows a reciever
to reject an incoming session.
Design goals.
-------------
o Unconfirmed service without negotiation.
SCP allows data to be sent with the session establishment; the
recepient does not confirm successful connection establishment, but
may reject unsuccessful attempts. This simplifies the design of the
protocol, and removes the latency required for a confirmed operation.
o Low overhead
SCP has a fixed overhead of 8 bytes per segment. This overhead is
half the size of an IPNG address, and is only incurred once per
segment, instead of once per packet.
o Simple design
The session protocol should be simple enough to implement for a
single application.
Protocol Description
Header Format:
--------------
32 24 16 8 0
| | | | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0|0|0|0|S|F|R|P| SESSION ID |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SEGMENT LENGTH |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
. .
. DATA .
. .
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
S = SYN F = FIN R = RST P = PUSH
Protocol Operation:
-------------------
Session ID allocation.
Each session is allocated a session identifier. Session Identifiers below
1024 are reserved. Session IDs allocated by clients are even; those allocated
by servers, odd.
Session establishment.
A session is established by setting the SYN bit in the first message sent on
that channel.
Graceful release.
A session is ended by sending a message with the FIN bit set. Each end of a
connection may be closed independently.
Disgraceful release.
A session may be terminated by sending a message with the RST bit set. All
pending data for that session should be discarded
Message boundaries.
A message boundary is marked by sending a message with the PUSH bit set. The
boundary is set at the final octet in this message, including that octet.
------- =_aaaaaaaaaa0--
