Status: Approved-Draft2-20150825

================================================
Description of the message v02 protocol / format
================================================

This file was used during the design phase, but post-implementation,
it is replaced by the sr_post(7) manual page.

This file documents final conclusions/proposals, reasoning/debates 
goes elsewhere.

Messages posted include four parts:
topic: <version>.<type>.<src>(.<dir>.)*.<filename>
headers: series of key-value pairs, as per AMQP spec.
1st line (whitespace separated fields): <date stamp> <srcURL> <relURL><newline>
rest of body:


The message topic breaks down as follows:

	<version>.<type>.[varies by version].<dir>.<dir>.<dir>...

	<version>:
		exp -- initial version, deprecated (not covered in this document)
		v00 -- used for NURP & PAN-AM in 2013-2014. (not covered in this document)
		v01 -- 2015 version.
		v02 -- 2015 switched to AMQP headers for non-mandatory components

	<type>:
		adm  - change settings 
			´admin´, ´config´, etc...

		log  - report status of operations.

		notify - ´post´ but in exp and v00 versions. (not covered here.)

		post - announce or notify that a new product block is available.
	       		possible strings: post,ann(ounce), not(ify)
		
	<source>:

Rest of this document assumes version 2 (v02 topic):

breaks down to:

<date stamp>: date
	YYYYMMDDHHMMSS.<decimal> 

<srcURL> -- the base URL used to retrieve the data.

	options: Complete URL:

	sftp://afsiext@cmcdataserver/data/NRPDS/outputs/NRPDS_HiRes_000.gif

	in the case where the URL does not end with a path separator ('/'), 
        the src path is taken to be the complete source of the file to retrieve.


	Static URL:

	sftp://afsiext@cmcdataserver/  

	If the URL ends with a path separator ('/'), then the src URL is 
        considered a prefix for the variable part of the retrieval URL.


<relURL> -- The relative path from the current directory in which to 
  	place the file.
	
	Two cases based on the end being a path separator or not.

	case 1: NURP/GIF/	

	based on the current working directory of the downloading client, 
	create a subdirectory called URP, and within that, a subdirectory 
	called GIF will be created.  The file name will be taken from the 
	srcpath.

	if the srcpath ends in pathsep, then the relpath here will be 
	concatenated to the srcpath, forming the complete retrieval URL.

	case 2: NRP/GIF/mine.gif

	if the  srcpath ends in pathsep, then the relpath will be concatenated 
	to srcpath for form the complete retrieval URL.

	if the src path does not end in pathsep, then the src URL is taken 
	as complete, and the file is renamed on download according to the 
	specification (in this case, mine.gif)


AMQP provides HEADERS which are key/value pairs.


Describe what part of the URL is being announced:

parts=1,sz  
	-- fetch in a single part, of the given size in bytes
	
parts=<i|p>,<bsz>,<fzb>,<bno>,<remainder>
	-- multipart fetch.

        -- File Segment strategy::

		i - inplace (do not create temporary files, just lseek within file.)
		    may result in .srsig file being created?
		p - part files.  use .part files,  suffix fixed.
		    do not know which will be default.

	-- file segment strategy can be overridden by client. just a suggestion.
	-- analogous to rsync options: --inplace, --partial, 


<blocksize in bytes>: bsz
        the number of bytes in a block.
	checksums are calculated per block, so one post 

<filesize in blocks>: fzb
	the integer total number of blocks in the file
	FIXME: (including the last block or not?)
	if set to 1.

	
<block#>: bno
  	0 origin, the block number covered by this posting.

<remainder>: brem
	normally 0, on the last block, it remaining blocks in the file 
        to transfer.

	-- if (fzb=1 and brem=0) 
	       then bsz=fsz in bytes in bytes. 
	       -- entire files replaced.
	       -- this is the same as rsync's --whole-file mode.
		
sum=<algorithm>,<value>

	<algorithm>

        d - checksum the entire data
        n - checksum the file name
        <script> - checksum with a script, named <script>

		<script> should be ´registered´ in the switch network.
       			registered means that all downstream subscribers 
			can obtain the script to validate the checksum. 
			there needs to be a retrieval mechanism.

	<value> is the checksum value

flow=<flowid>
	an arbitrary tag used for tracking of data through the network.

The two paths are subtly inter-related.  Neither can be interpreted on their own.  One must consider both path components.


FIXME: verify the following:
	fsz = Size of a file in bytes = ( bsz * (fsb-1) ) + brem ?


example 1:

v02.post.ec_cmc.NRDPS.GIF.NRDPS_HiRes_000.gif
201506011357.345 sftp://afsiext@cmcdataserver/data/NRPDS/outputs/NRDPS_HiRes_000.gif 
HEADERS:
parts=1,457
rename=NRDPS/GIF/ 
sum=d,<md5sum>
flow=exp13

	v01 - version of protocol
	post - indicates the type of message 

	version and type together determine format of following topics and the message body.

	ec_cmc - the account used to issue the post (unique in a network).
  
	  -- file size is 457  (== file size)
	  -- d - checksum was calculated on the body.
	  -- flow is called ´exp13´ by the poster...
	  -- complete source URL specified (does not end in '/')
	  -- relative path specified for 

	pull from: 
		sftp://afsiext@cmcdataserver/data/NRPDS/outputs/NRDPS_HiRes_000.gif

	complete relative download path:
		NRDPS/GIF/NRDPS_HiRes_000.gif

		-- takes file name from srcpath.
		-- may be modified by validation process.


example 2:

v02.post.ec_cmc.NRDPS.GIF.NRDPS_HiRes_000.gif
201506011357.345 http://afsiext@cmcdataserver/data/  
HEADERS:
rename=NRDPS/GIF/NRDPS_HiRes_000.gif
parts=1,457
sum=d,<md5sum>
flow=exp13

in this case, the
	pull from: 
		http://afsiext@cmcdataserver/data/NRPDS/GIF/NRDPS_HiRes_000.gif

		-- srcpath ends in '/', so concatenated, takes file from relative URL.
		-- true 'mirror'


	complete relative download path:
		NRDPS/GIF/NRDPS_HiRes_000.gif

		-- may be modified by validation process.

example 3:

v02.post.ec_cmc.NRDPS.GIF.NRDPS_HiRes_000.gif
201506011357.345 http://afsiext@cmcdataserver/data/ 
HEADERS:
rename=NRDPS/GIF/NRDPS_HiRes_000.gif
parts=i,457,0,0,1,0
sum=d,<md5sum>
flow=exp13

wait case.

wait=on/off