User story for PutFile using an offline pillar with extended feedback
This scenario describes a possible sequence of actions that lead to a file being transferred to a pillar based on off-line storage.
Note that there are no principal reason that hinders uploads to a pillar based on off-line media to follow an ordinary upload sequence. This scenario should therefore be regarded as an example of how optional parameters and messages may be useful in this particular situation.
To limit the scope of the scenario, it is assumed that the file is uploaded to only one pillar. Again, this is not a limitation in protocol or principle. It is just a simplification.
Technical background for the scenario
One convenient way of implementing a pillar based on off-line media is, to establish a “frontend” system that temporary buffers data to/from clients.
When data are ingested, buffering can be used to introduce a delay between reception and creating preservation media. This can be relevant, if large amount of data are received faster than data can be written to media. But in day-to-day operation it is probably much more relevant to delay writing files to media until a decent amount of data has been assembled in the buffer. (The reason for this is that the cost of manual operation involved in creating and storing physical media usually is many times higher than the cost of automatically receive files.)
The temporary buffering of data, create a situation where reception of the file and storing it on media are two different incidents in time that could possibly differ by weeks. The PutFile protocol has optional elements that can be used to ensure integrity across time.
User want to preserve a file using a single pillar SLA, where the Pillar used is based on an off-line medium.
The first part of this PutFile operation will at first resemble an ordinary (on-line media) PutFile operation, from contacting pillars until data is received by the pillar.
The PutFile conversation also starts out in the same way as an ordinary simple PutFile operation.
(File ready to upload on https)
Receive file through https
Optional: A pillar could acknowledge the reception of a GetFileRequest with a GetFileResponse with a “OK” value in the ResponseCode parameter. This indicates to the client, that the command is received, and the pillar expect the PutFile to end successfully. This is an optional step that could be convenient to trace every step of a PutFile operation.
The first real difference is this response. At this point the file is stored in a buffer, and has yet to reach permanent media. The pillar cannot respond with a PutFileComplete(), as this would indicate to the Client, that the PutFile operation is finished successfully, including writing the file to a medium. Instead the pillar responds with a GetFileResponse(ResponseCode=”OK_BF”) message, where the “OK_BF” code indicates, that the file has safely reached a temporary medium, in this case, the frontend systems buffer.
When the file is written to final storage medium, the Pillar can indicate this to the client, by sending a GetFileComplete() message. This indicates to the client, that as seen from the perspective of the Pillar, the PutFile is finished
Note that several weeks may pass between the GetFileResponse(ResponseCode=”OK_BF”) and the GetFileComplete() commands. One consequence of the large time delay is that the queue intended for transmission of the PutFileComplete message may not be available any more. (It would probably stress the definition of a *temporary* queue to survive for maybe several weeks.)
If a “Default ClientQueue” is available on the SLA, a pillar should use it in this situaton,