HTTPS as the underlying protocol
The protocol used for data exchange should meet the following properties:
- 8-bit clean - to maximize throughput
- Simple to handle in firewalls
- Have widespread support
- Support encryption/PKI
- Support resumption/partial transfer of large files
All those requirements are met by secure HTTP / HTTPS
Application of HTTPS to data exchange
When transferring data back and forth between clients and pillars, all data exchange communcation is initiated by a pillar. This requirement is introduced for two reasons: The first reason is that the pillar machinery is supposed to be well stuck behind possibly NAT 'ing firewalls, making communication very cumbersome. The other reason is security: All listening IP ports on a server introduce potential points of intrusion. Opening an outgoing connection is not without risks either, but the risk will be limited to the actual duration of the transfer. The requirement does have implications for data exchange, as pillar to pillar transfers have to be indirect.
All communication *may* be encrypted by server and client certificates (remember that the pillar is the HTTP client). In that case authenticity is guaranted. Unencrypted transfers may have their uses, though - ie. transferring data for presentation systems or simple harvesting of collections.
The HTTP method GET is used for transferring data from client to pillar, while PUT is used for transferring data from pillar to client. This means that a "Put" message exchange will actually result in an http GET and vice versa.
Partial transfers are supported through the HTTP "Range" header. Using this feature may be desirable when only some segment of a file is needed - ie for streaming video - or for dividing the transfer of large files into more manageable chunks.
At first it may seem contradictory that messaging is initiated by the clients, while data transfers are initiated by a pillar. On the network layer, however, the pillar will always be the initiator of communications, not the target, as connections between pillar and queueing system are also initiated by the piller.