{ "cells": [ { "cell_type": "markdown", "id": "chubby-tenant", "metadata": {}, "source": [ "# Downloading Using the Command Line\n", "\n", "This [jupyter notebook](https://jupyter.org) introduces [Sarracenia version 3](https://metpx.github.io/sarracenia) usage from the command line (mostly on Linux, but should be similar on Windows and Mac also, main difference being different conventions for where preferences and logs are stored.) This is probably the easiest way to work with Sarracenia. You configure a flow to download files into a directory, and you can read the directory to process the files there.\n" ] }, { "cell_type": "code", "execution_count": 1, "id": "neither-shannon", "metadata": {}, "outputs": [], "source": [ "import sarracenia\n", "!mkdir -p ~/.config/sr3/subscribe\n", "!mkdir -p ~/.cache/sr3/log" ] }, { "cell_type": "markdown", "id": "varying-armor", "metadata": { "jp-MarkdownHeadingCollapsed": true }, "source": [ "\n", "## Prerequisites\n", "\n", "The above is just a way to get jupyter notebooks to install metpx-sr3 on a server.\n", "Creating some directories in case people use API access without running things through the API. The basic pre-requisite is to have metpx-sr3 installed somehow, either as a .deb package, or using pip (or pip3) available to the environment used by jupyter.\n", "\n", "The rest of this notebook assumes [metpx-sr3](https://metpx.github.io/sarracenia) is installed." ] }, { "cell_type": "markdown", "id": "absolute-integral", "metadata": {}, "source": [ "## SR3\n", "\n", "The command line interface is called [sr3](../Reference/sr3.1.rst) (short for Sarracenia version 3). One defines\n", "flows to run using configuration files in a simple format: _keyword_ _value_ format.\n", "There are example configurations to get you started:" ] }, { "cell_type": "code", "execution_count": 2, "id": "drawn-opposition", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2025-05-06 16:01:03,837 402027 [ERROR] sarracenia.config _parse_binding ./shovel_dual_amis_feed_mqtt.conf:24 broker needed before subtopic\n", "2025-05-06 16:01:03,837 402027 [WARNING] sarracenia.config parse_line ./shovel_dual_amis_feed_mqtt.conf:26 broker needs to appear before *subtopic* unless you need different queues to have different settings\n", "Sample Configurations: (from: /home/peter/Sarracenia/sr3/sarracenia/examples )\n", "cpump/cno_trouble_f00.inc flow/amserver.conf \n", "flow/bc_trans.conf flow/opg.conf \n", "flow/poll.inc flow/post.inc \n", "flow/report.inc flow/sarra.inc \n", "flow/scheduled_aviation_wind_fax_charts.conf flow/sender.inc \n", "flow/shovel.inc flow/subscribe.inc \n", "flow/watch.inc flow/winnow.inc \n", "poll/airnow.conf poll/aws-nexrad.conf \n", "poll/copernicus_odata.conf poll/mail.conf \n", "poll/nasa-mls-nrt.conf poll/nasa_cmr_opendap.conf \n", "poll/nasa_cmr_other.conf poll/nasa_cmr_podaac.conf \n", "poll/noaa.conf poll/soapshc.conf \n", "poll/usgs.conf post/WMO_mesh_post.conf \n", "sarra/bc_trans.conf sarra/wmo_mesh.conf \n", "sender/am_send.conf sender/ec2collab.conf \n", "sender/pitcher_push.conf shovel/no_trouble_f00.inc \n", "subscribe/aws-nexrad.conf subscribe/dd_2mqtt.conf \n", "subscribe/dd_all.conf subscribe/dd_amis.conf \n", "subscribe/dd_aqhi.conf subscribe/dd_cacn_bulletins.conf \n", "subscribe/dd_citypage.conf subscribe/dd_cmml.conf \n", "subscribe/dd_gdps.conf subscribe/dd_radar.conf \n", "subscribe/dd_rdps.conf subscribe/dd_swob.conf \n", "subscribe/ddc_cap-xml.conf subscribe/ddc_normal.conf \n", "subscribe/download_all_nasa_earthdata.conf subscribe/downloademail.conf \n", "subscribe/dual_amis.conf subscribe/ec_ninjo-a.conf \n", "subscribe/get_copernicus.conf subscribe/hpfxWIS2DownloadAll.conf \n", "subscribe/hpfx_amis.conf subscribe/hpfx_citypage.conf \n", "subscribe/local_sub.conf subscribe/ping.conf \n", "subscribe/pitcher_pull.conf subscribe/sci2ec.conf \n", "subscribe/subnoaa.conf subscribe/subsoapshc.conf \n", "subscribe/subusgs.conf watch/master.conf \n", "watch/pitcher_client.conf watch/pitcher_server.conf \n", "watch/sci2ec.conf \n" ] } ], "source": [ "!sr3 list examples" ] }, { "cell_type": "markdown", "id": "affecting-marking", "metadata": {}, "source": [ "There are different kinds for flows: the examples are classified by flow type (poll, post, sarra, sender, shovel, etc.)\n", "A _subscribe_ is used by clients to download from a data pump. Let's pick one of those." ] }, { "cell_type": "code", "execution_count": 4, "id": "egyptian-suicide", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2025-05-06 16:02:58,977 402310 [WARNING] sarracenia.config parse_line ./dual_amis.conf:20 broker needs to appear before *subtopic* unless you need different queues to have different settings\n", "2025-05-06 16:02:58,977 402310 [ERROR] sarracenia.config _parse_binding ./shovel_dual_amis_feed_mqtt.conf:24 broker needed before subtopic\n", "2025-05-06 16:02:58,977 402310 [WARNING] sarracenia.config parse_line ./shovel_dual_amis_feed_mqtt.conf:26 broker needs to appear before *subtopic* unless you need different queues to have different settings\n", "add: 2025-05-06 16:02:58,978 402310 [INFO] root add matched existing ['subscribe/dual_amis']\n", "2025-05-06 16:02:58,978 402310 [ERROR] root add nothing specified to add\n", "\n" ] } ], "source": [ "!sr3 add subscribe/dual_amis.conf" ] }, { "cell_type": "markdown", "id": "d1179254-ac4c-49f8-b1a8-30f96d09290b", "metadata": {}, "source": [ "The files that are active for you are placed in ~/.config/sr3/\\/config_name. You can browse there \n", "and modify them with an editor if you like. You can also do that with _sr3 edit subscribe/hpfx_amis.conf_.\n", "\n", " # this is a feed of wmo bulletin (a set called AMIS in the old times)\n", "\n", " broker amqps://hpfx.collab.science.gc.ca/\n", " exchange xpublic\n", "\n", " # instances: number of downloading processes to run at once. Defaults to 1. Not enough for this case\n", " instances 5\n", " \n", " # expire, in operational use, should be longer than longest expected interruption\n", " expire 10m\n", "\n", " topicPrefix v02.post\n", " subtopic *.WXO-DD.bulletins.alphanumeric.#\n", " mirror false\n", " directory /tmp/hpfx_amis/\n", "\n", "Add the messageCountMax, so it doesn't run forever:" ] }, { "cell_type": "code", "execution_count": 5, "id": "primary-score", "metadata": {}, "outputs": [], "source": [ "!mkdir /tmp/dual_amis\n", "!echo messageCountMax 10 >>~/.config/sr3/subscribe/dual_amis.conf" ] }, { "cell_type": "markdown", "id": "ancient-scholarship", "metadata": {}, "source": [ "The root directory where files are to be placed needs to exist before you start.\n", "The above commands are to configure on a Linux machine, you might need something else on a mac or windows.\n", "\n", "You can then run a flow interactively with the _foreground_ action, and it will end quickly, like so:" ] }, { "cell_type": "code", "execution_count": 6, "id": "nominated-nerve", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2025-05-06 16:03:19,980 402343 [WARNING] sarracenia.config parse_line ./dual_amis.conf:20 broker needs to appear before *subtopic* unless you need different queues to have different settings\n", "2025-05-06 16:03:19,987 402343 [INFO] sarracenia.config finalize subscribe/dual_amis.conf overriding batch for consistency with messageCountMax: 10\n", "2025-05-06 16:03:19,988 402343 [ERROR] sarracenia.config _parse_binding ./shovel_dual_amis_feed_mqtt.conf:24 broker needed before subtopic\n", "2025-05-06 16:03:19,988 402343 [WARNING] sarracenia.config parse_line ./shovel_dual_amis_feed_mqtt.conf:26 broker needs to appear before *subtopic* unless you need different queues to have different settings\n", "2025-05-06 16:03:19,988 402343 [ERROR] sarracenia.config.subscription write failed: /home/peter/.cache/sr3/subscribe/shovel_dual_amis_feed_mqtt/subscriptions.json: [Errno 2] No such file or directory: '/home/peter/.cache/sr3/subscribe/shovel_dual_amis_feed_mqtt/subscriptions.json'\n", ".2025-05-06 16:03:20,379 [WARNING] 402349 sarracenia.config parse_line ./dual_amis.conf:20 broker needs to appear before *subtopic* unless you need different queues to have different settings\n", "2025-05-06 16:03:20,381 [INFO] 402349 sarracenia.config finalize subscribe/dual_amis overriding batch for consistency with messageCountMax: 10\n", "2025-05-06 16:03:20,382 [WARNING] 402349 sarracenia.config parse_line ./dual_amis.conf:20 broker needs to appear before *subtopic* unless you need different queues to have different settings\n", "2025-05-06 16:03:20,386 [INFO] 402349 sarracenia.config finalize subscribe/dual_amis overriding batch for consistency with messageCountMax: 10\n", "2025-05-06 16:03:20,623 [INFO] 402349 sarracenia.moth.amqp _queueDeclare queue declared q_anonymous.subscribe.dual_amis.peter_fractal_91605071 (as: amqps://anonymous@dd.weather.gc.ca/), (messages waiting: 0)\n", "2025-05-06 16:03:20,623 [INFO] 402349 sarracenia.moth.amqp getSetup binding q_anonymous.subscribe.dual_amis.peter_fractal_91605071 with v02.post.*.WXO-DD.bulletins.alphanumeric.# to xpublic (as: amqps://anonymous@dd.weather.gc.ca/)\n", "2025-05-06 16:03:20,911 [INFO] 402349 sarracenia.moth.amqp _queueDeclare queue declared q_anonymous.subscribe.dual_amis.peter_fractal_21908331 (as: amqps://anonymous@hpfx.collab.science.gc.ca/), (messages waiting: 0)\n", "2025-05-06 16:03:20,911 [INFO] 402349 sarracenia.moth.amqp getSetup binding q_anonymous.subscribe.dual_amis.peter_fractal_21908331 with v02.post.*.WXO-DD.bulletins.alphanumeric.# to xpublic (as: amqps://anonymous@hpfx.collab.science.gc.ca/)\n", "2025-05-06 16:03:21,420 [INFO] 402349 sarracenia.flowcb.log after_accept accepted: (lag: 1.08 ) exchange: xpublic subtopic: 20250506.WXO-DD.bulletins.alphanumeric.20250506.SR.KWAL.20 a file with baseUrl: https://hpfx.collab.science.gc.ca relPath: /20250506/WXO-DD/bulletins/alphanumeric/20250506/SR/KWAL/20/SRCN40_KWAL_062002___46610 sundew_extension: cvt_nws_bulletins-sr:KWAL:SR:3:Direct:20250506200318 id: wXFihSI size: 304 \n", "2025-05-06 16:03:21,560 [INFO] 402349 sarracenia.flowcb.log after_work downloaded ok: /tmp/dual_amis/SRCN40_KWAL_062002___46610 \n", "2025-05-06 16:03:21,701 [INFO] 402349 sarracenia.flowcb.log after_accept accepted: (lag: 0.33 ) exchange: xpublic subtopic: 20250506.WXO-DD.bulletins.alphanumeric.20250506.SX.KWAL.20 a file with baseUrl: https://dd2.weather.gc.ca/ relPath: 20250506/WXO-DD/bulletins/alphanumeric/20250506/SX/KWAL/20/SXCN40_KWAL_062002___61390 sundew_extension: cvt_nws_bulletins-sr:KWAL:SX:3:Direct:20250506200319 id: Pi7l0QK size: 197 \n", "2025-05-06 16:03:21,701 [INFO] 402349 sarracenia.flowcb.log after_accept accepted: (lag: 0.91 ) exchange: xpublic subtopic: 20250506.WXO-DD.bulletins.alphanumeric.20250506.SR.KWAL.20 a file with baseUrl: https://dd2.weather.gc.ca/ relPath: 20250506/WXO-DD/bulletins/alphanumeric/20250506/SR/KWAL/20/SRCN40_KWAL_062002___54152 sundew_extension: cvt_nws_bulletins-sr:KWAL:SR:3:Direct:20250506200319 id: yjsDp9L size: 220 \n", "2025-05-06 16:03:21,701 [INFO] 402349 sarracenia.flowcb.log after_accept accepted: (lag: 0.91 ) exchange: xpublic subtopic: 20250506.WXO-DD.bulletins.alphanumeric.20250506.SR.KWAL.20 a file with baseUrl: https://hpfx.collab.science.gc.ca relPath: /20250506/WXO-DD/bulletins/alphanumeric/20250506/SR/KWAL/20/SRCN40_KWAL_062002___54152 sundew_extension: cvt_nws_bulletins-sr:KWAL:SR:3:Direct:20250506200319 id: yjsDp9L size: 220 \n", "2025-05-06 16:03:21,701 [INFO] 402349 sarracenia.flowcb.log after_accept accepted: (lag: 0.91 ) exchange: xpublic subtopic: 20250506.WXO-DD.bulletins.alphanumeric.20250506.SX.KWAL.20 a file with baseUrl: https://hpfx.collab.science.gc.ca relPath: /20250506/WXO-DD/bulletins/alphanumeric/20250506/SX/KWAL/20/SXCN40_KWAL_062002___43824 sundew_extension: cvt_nws_bulletins-sr:KWAL:SX:3:Direct:20250506200319 id: GKphUPi size: 128 \n", "2025-05-06 16:03:21,701 [INFO] 402349 sarracenia.flowcb.log after_accept accepted: (lag: 0.33 ) exchange: xpublic subtopic: 20250506.WXO-DD.bulletins.alphanumeric.20250506.SX.KWAL.20 a file with baseUrl: https://hpfx.collab.science.gc.ca relPath: /20250506/WXO-DD/bulletins/alphanumeric/20250506/SX/KWAL/20/SXCN40_KWAL_062002___61390 sundew_extension: cvt_nws_bulletins-sr:KWAL:SX:3:Direct:20250506200319 id: Pi7l0QK size: 197 \n", "2025-05-06 16:03:21,701 [INFO] 402349 sarracenia.flowcb.log after_accept accepted: (lag: 0.32 ) exchange: xpublic subtopic: 20250506.WXO-DD.bulletins.alphanumeric.20250506.SR.KWAL.20 a file with baseUrl: https://hpfx.collab.science.gc.ca relPath: /20250506/WXO-DD/bulletins/alphanumeric/20250506/SR/KWAL/20/SRCN40_KWAL_062002___38624 sundew_extension: cvt_nws_bulletins-sr:KWAL:SR:3:Direct:20250506200319 id: ptS5f1m size: 442 \n", "2025-05-06 16:03:22,053 [INFO] 402349 sarracenia.flowcb.log after_work rejected: 406 mtime not newer /tmp/dual_amis/SRCN40_KWAL_062002___54152 \n", "2025-05-06 16:03:22,053 [INFO] 402349 sarracenia.flowcb.log after_work rejected: 406 mtime not newer /tmp/dual_amis/SXCN40_KWAL_062002___61390 \n", "2025-05-06 16:03:22,053 [INFO] 402349 sarracenia.flowcb.log after_work downloaded ok: /tmp/dual_amis/SXCN40_KWAL_062002___61390 \n", "2025-05-06 16:03:22,053 [INFO] 402349 sarracenia.flowcb.log after_work downloaded ok: /tmp/dual_amis/SRCN40_KWAL_062002___54152 \n", "2025-05-06 16:03:22,053 [INFO] 402349 sarracenia.flowcb.log after_work downloaded ok: /tmp/dual_amis/SXCN40_KWAL_062002___43824 \n", "2025-05-06 16:03:22,053 [INFO] 402349 sarracenia.flowcb.log after_work downloaded ok: /tmp/dual_amis/SRCN40_KWAL_062002___38624 \n", "2025-05-06 16:03:22,254 [INFO] 402349 sarracenia.flowcb.log after_accept accepted: (lag: 0.87 ) exchange: xpublic subtopic: 20250506.WXO-DD.bulletins.alphanumeric.20250506.SR.KWAL.20 a file with baseUrl: https://dd2.weather.gc.ca/ relPath: 20250506/WXO-DD/bulletins/alphanumeric/20250506/SR/KWAL/20/SRCN40_KWAL_062002___38624 sundew_extension: cvt_nws_bulletins-sr:KWAL:SR:3:Direct:20250506200319 id: ptS5f1m size: 442 \n", "2025-05-06 16:03:22,254 [INFO] 402349 sarracenia.flowcb.log after_accept accepted: (lag: 1.91 ) exchange: xpublic subtopic: 20250506.WXO-DD.bulletins.alphanumeric.20250506.SR.KWAL.20 a file with baseUrl: https://dd2.weather.gc.ca/ relPath: 20250506/WXO-DD/bulletins/alphanumeric/20250506/SR/KWAL/20/SRCN40_KWAL_062002___46610 sundew_extension: cvt_nws_bulletins-sr:KWAL:SR:3:Direct:20250506200318 id: wXFihSI size: 304 \n", "2025-05-06 16:03:22,254 [INFO] 402349 sarracenia.flowcb.log after_accept accepted: (lag: 1.45 ) exchange: xpublic subtopic: 20250506.WXO-DD.bulletins.alphanumeric.20250506.SX.KWAL.20 a file with baseUrl: https://dd1.weather.gc.ca/ relPath: 20250506/WXO-DD/bulletins/alphanumeric/20250506/SX/KWAL/20/SXCN40_KWAL_062002___39285 sundew_extension: cvt_nws_bulletins-sr:KWAL:SX:3:Direct:20250506200319 id: qeL+vwU size: 171 \n", "2025-05-06 16:03:22,254 [INFO] 402349 sarracenia.flowcb.log after_accept accepted: (lag: 0.53 ) exchange: xpublic subtopic: 20250506.WXO-DD.bulletins.alphanumeric.20250506.SX.KWAL.20 a file with baseUrl: https://dd1.weather.gc.ca/ relPath: 20250506/WXO-DD/bulletins/alphanumeric/20250506/SX/KWAL/20/SXAK50_KWAL_062002___5577 sundew_extension: cvt_nws_bulletins-sr:KWAL:SX:3:Direct:20250506200319 id: Jf/H/VF size: 174 \n", "2025-05-06 16:03:22,254 [INFO] 402349 sarracenia.flowcb.log after_accept accepted: (lag: 0.45 ) exchange: xpublic subtopic: 20250506.WXO-DD.bulletins.alphanumeric.20250506.SR.KWAL.20 a file with baseUrl: https://dd1.weather.gc.ca/ relPath: 20250506/WXO-DD/bulletins/alphanumeric/20250506/SR/KWAL/20/SRCN40_KWAL_062002___7734 sundew_extension: cvt_nws_bulletins-sr:KWAL:SR:3:Direct:20250506200320 id: rTGcmnm size: 147 \n", "2025-05-06 16:03:22,254 [INFO] 402349 sarracenia.flowcb.log after_accept accepted: (lag: 1.45 ) exchange: xpublic subtopic: 20250506.WXO-DD.bulletins.alphanumeric.20250506.WS.RJTD.20 a file with baseUrl: https://hpfx.collab.science.gc.ca relPath: /20250506/WXO-DD/bulletins/alphanumeric/20250506/WS/RJTD/20/WSJP31_RJTD_062002___18004 sundew_extension: cvt_nws_bulletins-sr:RJTD:WS:3:Direct:20250506200319 id: 5pFG06W size: 223 \n", "2025-05-06 16:03:22,254 [INFO] 402349 sarracenia.flowcb.log after_accept accepted: (lag: 1.45 ) exchange: xpublic subtopic: 20250506.WXO-DD.bulletins.alphanumeric.20250506.SX.KWAL.20 a file with baseUrl: https://hpfx.collab.science.gc.ca relPath: /20250506/WXO-DD/bulletins/alphanumeric/20250506/SX/KWAL/20/SXCN40_KWAL_062002___39285 sundew_extension: cvt_nws_bulletins-sr:KWAL:SX:3:Direct:20250506200319 id: qeL+vwU size: 171 \n", "2025-05-06 16:03:22,254 [INFO] 402349 sarracenia.flowcb.log after_accept accepted: (lag: 0.45 ) exchange: xpublic subtopic: 20250506.WXO-DD.bulletins.alphanumeric.20250506.SR.KWAL.20 a file with baseUrl: https://hpfx.collab.science.gc.ca relPath: /20250506/WXO-DD/bulletins/alphanumeric/20250506/SR/KWAL/20/SRWA20_KWAL_062002___50635 sundew_extension: cvt_nws_bulletins-sr:KWAL:SR:3:Direct:20250506200320 id: mzneyRw size: 163 \n", "2025-05-06 16:03:22,254 [INFO] 402349 sarracenia.flowcb.log after_accept accepted: (lag: 0.45 ) exchange: xpublic subtopic: 20250506.WXO-DD.bulletins.alphanumeric.20250506.SR.KWAL.20 a file with baseUrl: https://hpfx.collab.science.gc.ca relPath: /20250506/WXO-DD/bulletins/alphanumeric/20250506/SR/KWAL/20/SRCN40_KWAL_062002___7734 sundew_extension: cvt_nws_bulletins-sr:KWAL:SR:3:Direct:20250506200320 id: rTGcmnm size: 147 \n", "2025-05-06 16:03:22,254 [INFO] 402349 sarracenia.flowcb.log after_accept accepted: (lag: 0.53 ) exchange: xpublic subtopic: 20250506.WXO-DD.bulletins.alphanumeric.20250506.SR.KWAL.20 a file with baseUrl: https://hpfx.collab.science.gc.ca relPath: /20250506/WXO-DD/bulletins/alphanumeric/20250506/SR/KWAL/20/SRCN40_KWAL_062002___26444 sundew_extension: cvt_nws_bulletins-sr:KWAL:SR:3:Direct:20250506200319 id: kz9EKHa size: 129 \n", "2025-05-06 16:03:22,254 [INFO] 402349 sarracenia.flowcb.log after_accept accepted: (lag: 0.53 ) exchange: xpublic subtopic: 20250506.WXO-DD.bulletins.alphanumeric.20250506.SX.KWAL.20 a file with baseUrl: https://hpfx.collab.science.gc.ca relPath: /20250506/WXO-DD/bulletins/alphanumeric/20250506/SX/KWAL/20/SXAK50_KWAL_062002___5577 sundew_extension: cvt_nws_bulletins-sr:KWAL:SX:3:Direct:20250506200319 id: Jf/H/VF size: 174 \n", "2025-05-06 16:03:22,733 [INFO] 402349 sarracenia.flowcb.log after_work rejected: 406 mtime not newer /tmp/dual_amis/SRCN40_KWAL_062002___38624 \n", "2025-05-06 16:03:22,733 [INFO] 402349 sarracenia.flowcb.log after_work rejected: 406 mtime not newer /tmp/dual_amis/SRCN40_KWAL_062002___46610 \n", "2025-05-06 16:03:22,733 [INFO] 402349 sarracenia.flowcb.log after_work rejected: 406 mtime not newer /tmp/dual_amis/SXCN40_KWAL_062002___39285 \n", "2025-05-06 16:03:22,733 [INFO] 402349 sarracenia.flowcb.log after_work rejected: 406 mtime not newer /tmp/dual_amis/SRCN40_KWAL_062002___7734 \n", "2025-05-06 16:03:22,733 [INFO] 402349 sarracenia.flowcb.log after_work rejected: 406 mtime not newer /tmp/dual_amis/SXAK50_KWAL_062002___5577 \n", "2025-05-06 16:03:22,733 [INFO] 402349 sarracenia.flowcb.log after_work downloaded ok: /tmp/dual_amis/SXCN40_KWAL_062002___39285 \n", "2025-05-06 16:03:22,733 [INFO] 402349 sarracenia.flowcb.log after_work downloaded ok: /tmp/dual_amis/SXAK50_KWAL_062002___5577 \n", "2025-05-06 16:03:22,733 [INFO] 402349 sarracenia.flowcb.log after_work downloaded ok: /tmp/dual_amis/SRCN40_KWAL_062002___7734 \n", "2025-05-06 16:03:22,733 [INFO] 402349 sarracenia.flowcb.log after_work downloaded ok: /tmp/dual_amis/WSJP31_RJTD_062002___18004 \n", "2025-05-06 16:03:22,733 [INFO] 402349 sarracenia.flowcb.log after_work downloaded ok: /tmp/dual_amis/SRWA20_KWAL_062002___50635 \n", "2025-05-06 16:03:22,733 [INFO] 402349 sarracenia.flowcb.log after_work downloaded ok: /tmp/dual_amis/SRCN40_KWAL_062002___26444 \n", "2025-05-06 16:03:22,733 [INFO] 402349 sarracenia.flow run 19 messages processed > messageCountMax 10\n", "2025-05-06 16:03:22,734 [INFO] 402349 sarracenia.flow please_stop asked to stop\n", "2025-05-06 16:03:22,734 [INFO] 402349 sarracenia.moth please_stop asked to stop\n", "2025-05-06 16:03:22,734 [INFO] 402349 sarracenia.moth please_stop asked to stop\n", "2025-05-06 16:03:22,734 [INFO] 402349 sarracenia.flow _runHousekeeping on_housekeeping pid: 402349 subscribe/dual_amis instance: 0\n", "2025-05-06 16:03:22,734 [INFO] 402349 sarracenia.flowcb.gather.message on_housekeeping from amqps://anonymous@dd.weather.gc.ca/ messages: good: 7 bad: 0 bytes: 912Bytes average: 130Bytes\n", "2025-05-06 16:03:22,734 [INFO] 402349 sarracenia.flowcb.gather.message on_housekeeping from amqps://anonymous@hpfx.collab.science.gc.ca/ messages: good: 11 bad: 0 bytes: 1.5KiB average: 138Bytes\n", "2025-05-06 16:03:22,734 [INFO] 402349 sarracenia.flowcb.housekeeping.resources on_housekeeping Current cpu_times: user=0.42 system=0.09\n", "2025-05-06 16:03:22,734 [INFO] 402349 sarracenia.flowcb.housekeeping.resources on_housekeeping Current mem usage: 125.2MiB, accumulating count (11 or 18/100 so far) before self-setting threshold\n", "2025-05-06 16:03:22,734 [INFO] 402349 sarracenia.flowcb.log stats version: 3.00.59rc1, started: 2 seconds ago, last_housekeeping: 2.3 seconds ago \n", "2025-05-06 16:03:22,734 [INFO] 402349 sarracenia.flowcb.log stats messages received: 25, accepted: 18, rejected: 7 rate accepted: 72.0% or 7.7 m/s\n", "2025-05-06 16:03:22,734 [INFO] 402349 sarracenia.flowcb.log stats files transferred: 11 bytes: 2.2KiB rate: 980Bytes/sec\n", "2025-05-06 16:03:22,734 [INFO] 402349 sarracenia.flowcb.log stats lag: average: 0.83, maximum: 1.91 \n", "2025-05-06 16:03:22,734 [INFO] 402349 sarracenia.flow run 19 messages processed > messageCountMax 10\n", "2025-05-06 16:03:22,734 [INFO] 402349 sarracenia.flow please_stop asked to stop\n", "2025-05-06 16:03:22,735 [INFO] 402349 sarracenia.moth please_stop asked to stop\n", "2025-05-06 16:03:22,735 [INFO] 402349 sarracenia.moth please_stop asked to stop\n", "2025-05-06 16:03:22,735 [INFO] 402349 sarracenia.flow _runHousekeeping on_housekeeping pid: 402349 subscribe/dual_amis instance: 0\n", "2025-05-06 16:03:22,735 [INFO] 402349 sarracenia.flowcb.gather.message on_housekeeping from amqps://anonymous@dd.weather.gc.ca/ messages: good: 0 bad: 0 bytes: 0Bytes average: 0Bytes\n", "2025-05-06 16:03:22,735 [INFO] 402349 sarracenia.flowcb.gather.message on_housekeeping from amqps://anonymous@hpfx.collab.science.gc.ca/ messages: good: 0 bad: 0 bytes: 0Bytes average: 0Bytes\n", "2025-05-06 16:03:22,735 [INFO] 402349 sarracenia.flowcb.housekeeping.resources on_housekeeping Current cpu_times: user=0.42 system=0.09\n", "2025-05-06 16:03:22,735 [INFO] 402349 sarracenia.flowcb.housekeeping.resources on_housekeeping Current mem usage: 125.2MiB, accumulating count (11 or 18/100 so far) before self-setting threshold\n", "2025-05-06 16:03:22,735 [INFO] 402349 sarracenia.flowcb.log stats version: 3.00.59rc1, started: 2 seconds ago, last_housekeeping: 0.0 seconds ago \n", "2025-05-06 16:03:22,735 [INFO] 402349 sarracenia.flowcb.log stats messages received: 0, accepted: 0, rejected: 0 rate accepted: 0.0% or 0.0 m/s\n", "2025-05-06 16:03:22,735 [INFO] 402349 sarracenia.flowcb.log stats files transferred: 0 bytes: 0Bytes rate: 0Bytes/sec\n", "2025-05-06 16:03:22,776 [INFO] 402349 sarracenia.flow close flow/close completed cleanly pid: 402349 subscribe/dual_amis instance: 0\n", "\n" ] } ], "source": [ "!sr3 foreground subscribe/dual_amis.conf" ] }, { "cell_type": "markdown", "id": "foreign-european", "metadata": {}, "source": [ "As you can see, it downloaded five files to /tmp/amis.\n", "The _foreground_ action is intended to help with debugging, rather than real operations." ] }, { "cell_type": "code", "execution_count": 7, "id": "split-writing", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2025-05-06 16:03:56,913 402441 [WARNING] sarracenia.config parse_line ./dual_amis.conf:20 broker needs to appear before *subtopic* unless you need different queues to have different settings\n", "2025-05-06 16:03:56,913 402441 [INFO] sarracenia.config finalize subscribe/dual_amis.conf overriding batch for consistency with messageCountMax: 10\n", "status: \n", "Component/Config Processes Rates\n", " State Run Retry Que Lag Last %rej messages Data \n", " ----- --- ----- --- --- ---- ---- -------- ---- \n", "subscribe/dual_amis stop 0/0 0 0 0.00s n/a 0.0% 0m/s 0B/s \n", " Total Running Configs: 0 ( Processes: 0 missing: 0 stray: 0 )\n", " Memory: uss:0Bytes rss:0Bytes vms:0Bytes \n", " CPU Time: User:0.00s System:0.00s \n", "\t Pub/Sub Received: 0m/s (0B/s), Sent: 0m/s (0B/s) Queued: 0 Retry: 0, Mean lag: 0.00s\n", "\t Data Received: 0f/s (0B/s), Sent: 0f/s (0B/s) \n" ] } ], "source": [ "!sr3 status" ] }, { "cell_type": "markdown", "id": "rocky-unemployment", "metadata": {}, "source": [ "Above, you can see there is 1 configuration in your list. You can have hundreds. The columns on the right refer to how many instances you have for each configuration. In the example above, _instances_ is set to 5, so one would expect to see 5 running instances when it would be running. You can start specifc configurations (in this case a subscribe config) with _sr3 start subscribe/\\_, or start all active configs from all components (sarra, subscribe, watch, winnow, etc.) with _sr3 start_" ] }, { "cell_type": "code", "execution_count": 8, "id": "neural-laugh", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2025-05-06 16:04:11,004 402457 [WARNING] sarracenia.config parse_line ./dual_amis.conf:20 broker needs to appear before *subtopic* unless you need different queues to have different settings\n", "2025-05-06 16:04:11,005 402457 [INFO] sarracenia.config finalize subscribe/dual_amis.conf overriding batch for consistency with messageCountMax: 10\n", "tail: cannot open '/home/peter/.cache/sr3/log/subscribe_dual_amis_01.log' for reading: No such file or directory\n", "tail: no files remaining\n", "2025-05-06 16:04:11,007 402457 [CRITICAL] root run_command subprocess.run failed err=Command '['tail', '-f', '/home/peter/.cache/sr3/log/subscribe_dual_amis_01.log']' returned non-zero exit status 1.\n", "\n" ] } ], "source": [ "!sr3 log subscribe/dual_amis.conf" ] }, { "cell_type": "markdown", "id": "leading-matthew", "metadata": {}, "source": [ "When running in the background, output needs to go a log file. Since we have only ran this configuration file in the foreground, asking to see the log prints an error about the log being missing. This tells you that the logs are in the _~/.cache/sr3/log_ directory. Logs can be monitored in real-time with traditional tools such as _tail -f_ or _grep_.\n", "\n", "_sr3 stop_ does what you expect.\n", "\n", "Processes can crash. In the _sr3 status_ output above, if the number of processes in the Run column is less than in the Exp (for Expected) one, then it means that some instances have crashed. You can repair it (just start the missing instances) with:\n", "\n", "_sr3 sanity_ -- start missing instances, also kill strays if any found.\n", "\n", "So that's it, an introduction to running configurations in Sarracenia from the command line.\n", "\n", "\n", "## Conclusion\n", "\n", "If all you want to do is obtain data from a data pump in real-time, the easiest way to go is using the command line interface to control some processes that run all the time so that they dump files in a certain directory.\n", "\n", "It isn't very efficient though. When dealing with a large number of files and aiming for high-speed processing, it’s more efficient to have your own application receive notifications about file arrivals rather than scanning a directory. This approach reduces CPU and I/O overhead while improving processing speed.\n", "\n", "The easiest way to do that is to add some callbacks to your flows. We'll cover that next." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.3" } }, "nbformat": 4, "nbformat_minor": 5 }