================================ How to setup a Remote Subscriber ================================ This example goes over how to subscribe to the swob files from the Environment Canada Weather office. Setup ~~~~~ Initialize the credentials storage in the `~/.config/sr3/credentials.conf` file:: $ sr3 edit credentials.conf amqps://anonymous:anonymous@dd.weather.gc.ca The format is a complete url on each line (`amqps://:@`). This credentials.conf file should be private (linux octal permissions: 0600). .conf files placed in the ``~/.config/sr3/subscribe_directory`` will be automatically found by ``subscribe``, rather than giving the full path. The *edit* command starts the user's configured editor on the file to be created, in the correct directory:: $ sr3 edit subscribe/swob.conf broker amqps://anonymous@dd.weather.gc.ca subtopic observations.swob-ml.# topicPrefix v02.post directory /tmp/swob_downloads accept .* $ mkdir /tmp/swob_downloads $ sr3 status subscribe/swob 2017-12-14 06:54:54,010 [INFO] subscribe swob 01 is stopped .. ERROR:: Currrently edit is failing if there isn't a file in the expected location (it does in fact, not create a file) See issue `#251 `_ for more info or to complain. In the interim instead use:: $ mkdir -p .config/sr3/subscribe $ touch $_/swob.conf $ sr3 edit swob.conf *broker* indicates where to connect to get the stream of notifications. The term *broker* is taken from AMQP (http://www.amqp.org), the protocol used to transfer the notifications. The notifications that will be received all have *topics* that correspond to their URL. .. NOTE:: Omitting ``directory`` from the config file will write the files in the present working directory. Given how quickly they arrive, be prepared to clean up. Startup ~~~~~~~ Now start up the newly created subscriber:: $ sr3 start swob 2015-12-03 06:53:35,268 [INFO] user_config = 0 ../swob.conf 2015-12-03 06:53:35,269 [INFO] instances 1 2015-12-03 06:53:35,270 [INFO] sr3 subscribe swob 0001 started Activity can be monitored via log files in ``~/.cache/sr3/log/`` or with the *log* command:: $ sr3 log swob 2015-12-03 06:53:35,635 [INFO] Binding queue q_anonymous.subscribe.swob.21096474.62787751 with key v02.post.observations.swob-ml.# to exchange xpublic on broker amqps://anonymous@dd.weather.gc.ca/ 2015-12-03 17:32:01,834 [INFO] user_config = 1 ../swob.conf 2015-12-03 17:32:01,835 [INFO] subscribe start 2015-12-03 17:32:01,835 [INFO] subscribe run 2015-12-03 17:32:01,835 [INFO] AMQP broker(dd.weather.gc.ca) user(anonymous) vhost(/) 2015-12-03 17:32:01,835 [INFO] AMQP input : exchange(xpublic) topic(v02.post.observations.swob-ml.#) 2015-12-03 17:32:01,835 [INFO] AMQP output: exchange(xs_anonymous) topic(v02.report.#) 2015-12-03 17:32:08,191 [INFO] Binding queue q_anonymous.subscribe.swob.21096474.62787751 with key v02.post.observations.swob-ml.# to exchange xpublic on broker amqps://anonymous@dd.weather.gc.ca/ ``[Ctrl] + [C]`` to exit watching the logs. The startup log appears normal, indicating the authentication information was accepted. ``Subscribe`` will get the notification and download the file into the present working directory (unless otherwise specified in the configuration file). ---- A normal download looks like this:: 2015-12-03 17:32:15,031 [INFO] Received topic v02.post.observations.swob-ml.20151203.CMED 2015-12-03 17:32:15,031 [INFO] Received notice 20151203223214.699 http://dd2.weather.gc.ca/observations/swob-ml/20151203/CMED/2015-12-03-2200-CMED-AUTO-swob.xml 2015-12-03 17:32:15,031 [INFO] Received headers {'filename': '2015-12-03-2200-CMED-AUTO-swob.xml', 'parts': '1,3738,1,0,0', 'sum': 'd,157a9e98406e38a8252eaadf68c0ed60', 'source': 'metpx', 'to_clusters': 'DD,DDI.CMC,DDI.ED M', 'from_cluster': 'DD'} 2015-12-03 17:32:15,031 [INFO] downloading/copying into ./2015-12-03-2200-CMED-AUTO-swob.xml Giving all the information contained in the notification. Here is a failure:: 2015-12-03 17:32:30,715 [INFO] Downloads: http://dd2.weather.gc.ca/observations/swob-ml/20151203/CXFB/2015-12-03-2200-CXFB-AUTO-swob.xml into ./2015-12-03-2200-CXFB-AUTO-swob.xml 0-6791 2015-12-03 17:32:30,786 [ERROR] Download failed http://dd2.weather.gc.ca/observations/swob-ml/20151203/CXFB/2015-12-03-2200-CXFB-AUTO-swob.xml 2015-12-03 17:32:30,787 [ERROR] Server couldn't fulfill the request. Error code: 404, Not Found This message is not always a failure as ``subscribe`` retries a few times before giving up. After a few minutes, here is what the download directory looks like:: $ ls -al | tail -rw-rw-rw- 1 peter peter 7875 Dec 3 17:36 2015-12-03-2236-CL3D-AUTO-minute-swob.xml -rw-rw-rw- 1 peter peter 7868 Dec 3 17:37 2015-12-03-2236-CL3G-AUTO-minute-swob.xml -rw-rw-rw- 1 peter peter 7022 Dec 3 17:37 2015-12-03-2236-CTRY-AUTO-minute-swob.xml -rw-rw-rw- 1 peter peter 6876 Dec 3 17:37 2015-12-03-2236-CYPY-AUTO-swob.xml -rw-rw-rw- 1 peter peter 6574 Dec 3 17:36 2015-12-03-2236-CYZP-AUTO-swob.xml -rw-rw-rw- 1 peter peter 7871 Dec 3 17:37 2015-12-03-2237-CL3D-AUTO-minute-swob.xml -rw-rw-rw- 1 peter peter 7873 Dec 3 17:37 2015-12-03-2237-CL3G-AUTO-minute-swob.xml -rw-rw-rw- 1 peter peter 7037 Dec 3 17:37 2015-12-03-2237-CTBF-AUTO-minute-swob.xml -rw-rw-rw- 1 peter peter 7022 Dec 3 17:37 2015-12-03-2237-CTRY-AUTO-minute-swob.xml -rw-rw-rw- 1 peter peter 122140 Dec 3 17:38 subscribe_dd_swob_0001.log Cleanup ~~~~~~~ To not download more files, stop the subscriber:: $ sr3 stop subscribe/swob 2015-12-03 17:32:22,219 [INFO] subscribe swob 01 stopped This however leaves the queue that ``sr3 start subscribe/swob`` setup on the broker active, as to allow a failed subscriber to attempt reconnecting without loosing progress. That is until the broker times out the queue and removes it. To tell the broker that we are finished with the queue, tell the subscriber to cleanup:: $ sr3 cleanup subscribe/swob 2015-12-03 17:32:22,008 [INFO] subscribe swob cleanup 2015-12-03 17:32:22,008 [INFO] AMQP broker(dd.weatheer.gc.ca) user(anonymous) vhost() 2015-12-03 17:32:22,008 [INFO] Using amqp module (AMQP 0-9-1) 2015-12-03 17:32:22,008 [INFO] deleting queue q_anonymous.subscribe.swob.21096474.62787751 (anonymous@dd.weather.gc.ca) Best practice is to clear the queue when done as to lessen the load on the broker.