Downloading Using the Command Line
This jupyter notebook introduces Sarracenia version 3 usage from the command line (mostly on Linux, but should be similar on Windows and Mac also, main difference being different conventions for where preferences and logs are stored.) This is probably the easiest way to work with Sarracenia. You configure a flow to download files into a directory, and you can read the directory to process the files there.
[1]:
import sarracenia
!mkdir -p ~/.config/sr3/subscribe
!mkdir -p ~/.cache/sr3/log
Prerequisites
The above is just a way to get jupyter notebooks to install metpx-sr3 on a server. Creating some directories in case people use API access without running things through the API. The basic pre-requisite is to have metpx-sr3 installed somehow, either as a .deb package, or using pip (or pip3) available to the environment used by jupyter.
The rest of this notebook assumes metpx-sr3 is installed.
SR3
The command line interface is called sr3 (short for Sarracenia version 3). One defines flows to run using configuration files in a simple format: keyword value format. There are example configurations to get you started:
[2]:
!sr3 list examples
2025-05-06 16:01:03,837 402027 [ERROR] sarracenia.config _parse_binding ./shovel_dual_amis_feed_mqtt.conf:24 broker needed before subtopic
2025-05-06 16:01:03,837 402027 [WARNING] sarracenia.config parse_line ./shovel_dual_amis_feed_mqtt.conf:26 broker needs to appear before *subtopic* unless you need different queues to have different settings
Sample Configurations: (from: /home/peter/Sarracenia/sr3/sarracenia/examples )
cpump/cno_trouble_f00.inc flow/amserver.conf
flow/bc_trans.conf flow/opg.conf
flow/poll.inc flow/post.inc
flow/report.inc flow/sarra.inc
flow/scheduled_aviation_wind_fax_charts.conf flow/sender.inc
flow/shovel.inc flow/subscribe.inc
flow/watch.inc flow/winnow.inc
poll/airnow.conf poll/aws-nexrad.conf
poll/copernicus_odata.conf poll/mail.conf
poll/nasa-mls-nrt.conf poll/nasa_cmr_opendap.conf
poll/nasa_cmr_other.conf poll/nasa_cmr_podaac.conf
poll/noaa.conf poll/soapshc.conf
poll/usgs.conf post/WMO_mesh_post.conf
sarra/bc_trans.conf sarra/wmo_mesh.conf
sender/am_send.conf sender/ec2collab.conf
sender/pitcher_push.conf shovel/no_trouble_f00.inc
subscribe/aws-nexrad.conf subscribe/dd_2mqtt.conf
subscribe/dd_all.conf subscribe/dd_amis.conf
subscribe/dd_aqhi.conf subscribe/dd_cacn_bulletins.conf
subscribe/dd_citypage.conf subscribe/dd_cmml.conf
subscribe/dd_gdps.conf subscribe/dd_radar.conf
subscribe/dd_rdps.conf subscribe/dd_swob.conf
subscribe/ddc_cap-xml.conf subscribe/ddc_normal.conf
subscribe/download_all_nasa_earthdata.conf subscribe/downloademail.conf
subscribe/dual_amis.conf subscribe/ec_ninjo-a.conf
subscribe/get_copernicus.conf subscribe/hpfxWIS2DownloadAll.conf
subscribe/hpfx_amis.conf subscribe/hpfx_citypage.conf
subscribe/local_sub.conf subscribe/ping.conf
subscribe/pitcher_pull.conf subscribe/sci2ec.conf
subscribe/subnoaa.conf subscribe/subsoapshc.conf
subscribe/subusgs.conf watch/master.conf
watch/pitcher_client.conf watch/pitcher_server.conf
watch/sci2ec.conf
There are different kinds for flows: the examples are classified by flow type (poll, post, sarra, sender, shovel, etc.) A subscribe is used by clients to download from a data pump. Let’s pick one of those.
[4]:
!sr3 add subscribe/dual_amis.conf
2025-05-06 16:02:58,977 402310 [WARNING] sarracenia.config parse_line ./dual_amis.conf:20 broker needs to appear before *subtopic* unless you need different queues to have different settings
2025-05-06 16:02:58,977 402310 [ERROR] sarracenia.config _parse_binding ./shovel_dual_amis_feed_mqtt.conf:24 broker needed before subtopic
2025-05-06 16:02:58,977 402310 [WARNING] sarracenia.config parse_line ./shovel_dual_amis_feed_mqtt.conf:26 broker needs to appear before *subtopic* unless you need different queues to have different settings
add: 2025-05-06 16:02:58,978 402310 [INFO] root add matched existing ['subscribe/dual_amis']
2025-05-06 16:02:58,978 402310 [ERROR] root add nothing specified to add
The files that are active for you are placed in ~/.config/sr3/<flow_type>/config_name. You can browse there and modify them with an editor if you like. You can also do that with sr3 edit subscribe/hpfx_amis.conf.
# this is a feed of wmo bulletin (a set called AMIS in the old times)
broker amqps://hpfx.collab.science.gc.ca/
exchange xpublic
# instances: number of downloading processes to run at once. Defaults to 1. Not enough for this case
instances 5
# expire, in operational use, should be longer than longest expected interruption
expire 10m
topicPrefix v02.post
subtopic *.WXO-DD.bulletins.alphanumeric.#
mirror false
directory /tmp/hpfx_amis/
Add the messageCountMax, so it doesn’t run forever:
[5]:
!mkdir /tmp/dual_amis
!echo messageCountMax 10 >>~/.config/sr3/subscribe/dual_amis.conf
The root directory where files are to be placed needs to exist before you start. The above commands are to configure on a Linux machine, you might need something else on a mac or windows.
You can then run a flow interactively with the foreground action, and it will end quickly, like so:
[6]:
!sr3 foreground subscribe/dual_amis.conf
2025-05-06 16:03:19,980 402343 [WARNING] sarracenia.config parse_line ./dual_amis.conf:20 broker needs to appear before *subtopic* unless you need different queues to have different settings
2025-05-06 16:03:19,987 402343 [INFO] sarracenia.config finalize subscribe/dual_amis.conf overriding batch for consistency with messageCountMax: 10
2025-05-06 16:03:19,988 402343 [ERROR] sarracenia.config _parse_binding ./shovel_dual_amis_feed_mqtt.conf:24 broker needed before subtopic
2025-05-06 16:03:19,988 402343 [WARNING] sarracenia.config parse_line ./shovel_dual_amis_feed_mqtt.conf:26 broker needs to appear before *subtopic* unless you need different queues to have different settings
2025-05-06 16:03:19,988 402343 [ERROR] sarracenia.config.subscription write failed: /home/peter/.cache/sr3/subscribe/shovel_dual_amis_feed_mqtt/subscriptions.json: [Errno 2] No such file or directory: '/home/peter/.cache/sr3/subscribe/shovel_dual_amis_feed_mqtt/subscriptions.json'
.2025-05-06 16:03:20,379 [WARNING] 402349 sarracenia.config parse_line ./dual_amis.conf:20 broker needs to appear before *subtopic* unless you need different queues to have different settings
2025-05-06 16:03:20,381 [INFO] 402349 sarracenia.config finalize subscribe/dual_amis overriding batch for consistency with messageCountMax: 10
2025-05-06 16:03:20,382 [WARNING] 402349 sarracenia.config parse_line ./dual_amis.conf:20 broker needs to appear before *subtopic* unless you need different queues to have different settings
2025-05-06 16:03:20,386 [INFO] 402349 sarracenia.config finalize subscribe/dual_amis overriding batch for consistency with messageCountMax: 10
2025-05-06 16:03:20,623 [INFO] 402349 sarracenia.moth.amqp _queueDeclare queue declared q_anonymous.subscribe.dual_amis.peter_fractal_91605071 (as: amqps://anonymous@dd.weather.gc.ca/), (messages waiting: 0)
2025-05-06 16:03:20,623 [INFO] 402349 sarracenia.moth.amqp getSetup binding q_anonymous.subscribe.dual_amis.peter_fractal_91605071 with v02.post.*.WXO-DD.bulletins.alphanumeric.# to xpublic (as: amqps://anonymous@dd.weather.gc.ca/)
2025-05-06 16:03:20,911 [INFO] 402349 sarracenia.moth.amqp _queueDeclare queue declared q_anonymous.subscribe.dual_amis.peter_fractal_21908331 (as: amqps://anonymous@hpfx.collab.science.gc.ca/), (messages waiting: 0)
2025-05-06 16:03:20,911 [INFO] 402349 sarracenia.moth.amqp getSetup binding q_anonymous.subscribe.dual_amis.peter_fractal_21908331 with v02.post.*.WXO-DD.bulletins.alphanumeric.# to xpublic (as: amqps://anonymous@hpfx.collab.science.gc.ca/)
2025-05-06 16:03:21,420 [INFO] 402349 sarracenia.flowcb.log after_accept accepted: (lag: 1.08 ) exchange: xpublic subtopic: 20250506.WXO-DD.bulletins.alphanumeric.20250506.SR.KWAL.20 a file with baseUrl: https://hpfx.collab.science.gc.ca relPath: /20250506/WXO-DD/bulletins/alphanumeric/20250506/SR/KWAL/20/SRCN40_KWAL_062002___46610 sundew_extension: cvt_nws_bulletins-sr:KWAL:SR:3:Direct:20250506200318 id: wXFihSI size: 304
2025-05-06 16:03:21,560 [INFO] 402349 sarracenia.flowcb.log after_work downloaded ok: /tmp/dual_amis/SRCN40_KWAL_062002___46610
2025-05-06 16:03:21,701 [INFO] 402349 sarracenia.flowcb.log after_accept accepted: (lag: 0.33 ) exchange: xpublic subtopic: 20250506.WXO-DD.bulletins.alphanumeric.20250506.SX.KWAL.20 a file with baseUrl: https://dd2.weather.gc.ca/ relPath: 20250506/WXO-DD/bulletins/alphanumeric/20250506/SX/KWAL/20/SXCN40_KWAL_062002___61390 sundew_extension: cvt_nws_bulletins-sr:KWAL:SX:3:Direct:20250506200319 id: Pi7l0QK size: 197
2025-05-06 16:03:21,701 [INFO] 402349 sarracenia.flowcb.log after_accept accepted: (lag: 0.91 ) exchange: xpublic subtopic: 20250506.WXO-DD.bulletins.alphanumeric.20250506.SR.KWAL.20 a file with baseUrl: https://dd2.weather.gc.ca/ relPath: 20250506/WXO-DD/bulletins/alphanumeric/20250506/SR/KWAL/20/SRCN40_KWAL_062002___54152 sundew_extension: cvt_nws_bulletins-sr:KWAL:SR:3:Direct:20250506200319 id: yjsDp9L size: 220
2025-05-06 16:03:21,701 [INFO] 402349 sarracenia.flowcb.log after_accept accepted: (lag: 0.91 ) exchange: xpublic subtopic: 20250506.WXO-DD.bulletins.alphanumeric.20250506.SR.KWAL.20 a file with baseUrl: https://hpfx.collab.science.gc.ca relPath: /20250506/WXO-DD/bulletins/alphanumeric/20250506/SR/KWAL/20/SRCN40_KWAL_062002___54152 sundew_extension: cvt_nws_bulletins-sr:KWAL:SR:3:Direct:20250506200319 id: yjsDp9L size: 220
2025-05-06 16:03:21,701 [INFO] 402349 sarracenia.flowcb.log after_accept accepted: (lag: 0.91 ) exchange: xpublic subtopic: 20250506.WXO-DD.bulletins.alphanumeric.20250506.SX.KWAL.20 a file with baseUrl: https://hpfx.collab.science.gc.ca relPath: /20250506/WXO-DD/bulletins/alphanumeric/20250506/SX/KWAL/20/SXCN40_KWAL_062002___43824 sundew_extension: cvt_nws_bulletins-sr:KWAL:SX:3:Direct:20250506200319 id: GKphUPi size: 128
2025-05-06 16:03:21,701 [INFO] 402349 sarracenia.flowcb.log after_accept accepted: (lag: 0.33 ) exchange: xpublic subtopic: 20250506.WXO-DD.bulletins.alphanumeric.20250506.SX.KWAL.20 a file with baseUrl: https://hpfx.collab.science.gc.ca relPath: /20250506/WXO-DD/bulletins/alphanumeric/20250506/SX/KWAL/20/SXCN40_KWAL_062002___61390 sundew_extension: cvt_nws_bulletins-sr:KWAL:SX:3:Direct:20250506200319 id: Pi7l0QK size: 197
2025-05-06 16:03:21,701 [INFO] 402349 sarracenia.flowcb.log after_accept accepted: (lag: 0.32 ) exchange: xpublic subtopic: 20250506.WXO-DD.bulletins.alphanumeric.20250506.SR.KWAL.20 a file with baseUrl: https://hpfx.collab.science.gc.ca relPath: /20250506/WXO-DD/bulletins/alphanumeric/20250506/SR/KWAL/20/SRCN40_KWAL_062002___38624 sundew_extension: cvt_nws_bulletins-sr:KWAL:SR:3:Direct:20250506200319 id: ptS5f1m size: 442
2025-05-06 16:03:22,053 [INFO] 402349 sarracenia.flowcb.log after_work rejected: 406 mtime not newer /tmp/dual_amis/SRCN40_KWAL_062002___54152
2025-05-06 16:03:22,053 [INFO] 402349 sarracenia.flowcb.log after_work rejected: 406 mtime not newer /tmp/dual_amis/SXCN40_KWAL_062002___61390
2025-05-06 16:03:22,053 [INFO] 402349 sarracenia.flowcb.log after_work downloaded ok: /tmp/dual_amis/SXCN40_KWAL_062002___61390
2025-05-06 16:03:22,053 [INFO] 402349 sarracenia.flowcb.log after_work downloaded ok: /tmp/dual_amis/SRCN40_KWAL_062002___54152
2025-05-06 16:03:22,053 [INFO] 402349 sarracenia.flowcb.log after_work downloaded ok: /tmp/dual_amis/SXCN40_KWAL_062002___43824
2025-05-06 16:03:22,053 [INFO] 402349 sarracenia.flowcb.log after_work downloaded ok: /tmp/dual_amis/SRCN40_KWAL_062002___38624
2025-05-06 16:03:22,254 [INFO] 402349 sarracenia.flowcb.log after_accept accepted: (lag: 0.87 ) exchange: xpublic subtopic: 20250506.WXO-DD.bulletins.alphanumeric.20250506.SR.KWAL.20 a file with baseUrl: https://dd2.weather.gc.ca/ relPath: 20250506/WXO-DD/bulletins/alphanumeric/20250506/SR/KWAL/20/SRCN40_KWAL_062002___38624 sundew_extension: cvt_nws_bulletins-sr:KWAL:SR:3:Direct:20250506200319 id: ptS5f1m size: 442
2025-05-06 16:03:22,254 [INFO] 402349 sarracenia.flowcb.log after_accept accepted: (lag: 1.91 ) exchange: xpublic subtopic: 20250506.WXO-DD.bulletins.alphanumeric.20250506.SR.KWAL.20 a file with baseUrl: https://dd2.weather.gc.ca/ relPath: 20250506/WXO-DD/bulletins/alphanumeric/20250506/SR/KWAL/20/SRCN40_KWAL_062002___46610 sundew_extension: cvt_nws_bulletins-sr:KWAL:SR:3:Direct:20250506200318 id: wXFihSI size: 304
2025-05-06 16:03:22,254 [INFO] 402349 sarracenia.flowcb.log after_accept accepted: (lag: 1.45 ) exchange: xpublic subtopic: 20250506.WXO-DD.bulletins.alphanumeric.20250506.SX.KWAL.20 a file with baseUrl: https://dd1.weather.gc.ca/ relPath: 20250506/WXO-DD/bulletins/alphanumeric/20250506/SX/KWAL/20/SXCN40_KWAL_062002___39285 sundew_extension: cvt_nws_bulletins-sr:KWAL:SX:3:Direct:20250506200319 id: qeL+vwU size: 171
2025-05-06 16:03:22,254 [INFO] 402349 sarracenia.flowcb.log after_accept accepted: (lag: 0.53 ) exchange: xpublic subtopic: 20250506.WXO-DD.bulletins.alphanumeric.20250506.SX.KWAL.20 a file with baseUrl: https://dd1.weather.gc.ca/ relPath: 20250506/WXO-DD/bulletins/alphanumeric/20250506/SX/KWAL/20/SXAK50_KWAL_062002___5577 sundew_extension: cvt_nws_bulletins-sr:KWAL:SX:3:Direct:20250506200319 id: Jf/H/VF size: 174
2025-05-06 16:03:22,254 [INFO] 402349 sarracenia.flowcb.log after_accept accepted: (lag: 0.45 ) exchange: xpublic subtopic: 20250506.WXO-DD.bulletins.alphanumeric.20250506.SR.KWAL.20 a file with baseUrl: https://dd1.weather.gc.ca/ relPath: 20250506/WXO-DD/bulletins/alphanumeric/20250506/SR/KWAL/20/SRCN40_KWAL_062002___7734 sundew_extension: cvt_nws_bulletins-sr:KWAL:SR:3:Direct:20250506200320 id: rTGcmnm size: 147
2025-05-06 16:03:22,254 [INFO] 402349 sarracenia.flowcb.log after_accept accepted: (lag: 1.45 ) exchange: xpublic subtopic: 20250506.WXO-DD.bulletins.alphanumeric.20250506.WS.RJTD.20 a file with baseUrl: https://hpfx.collab.science.gc.ca relPath: /20250506/WXO-DD/bulletins/alphanumeric/20250506/WS/RJTD/20/WSJP31_RJTD_062002___18004 sundew_extension: cvt_nws_bulletins-sr:RJTD:WS:3:Direct:20250506200319 id: 5pFG06W size: 223
2025-05-06 16:03:22,254 [INFO] 402349 sarracenia.flowcb.log after_accept accepted: (lag: 1.45 ) exchange: xpublic subtopic: 20250506.WXO-DD.bulletins.alphanumeric.20250506.SX.KWAL.20 a file with baseUrl: https://hpfx.collab.science.gc.ca relPath: /20250506/WXO-DD/bulletins/alphanumeric/20250506/SX/KWAL/20/SXCN40_KWAL_062002___39285 sundew_extension: cvt_nws_bulletins-sr:KWAL:SX:3:Direct:20250506200319 id: qeL+vwU size: 171
2025-05-06 16:03:22,254 [INFO] 402349 sarracenia.flowcb.log after_accept accepted: (lag: 0.45 ) exchange: xpublic subtopic: 20250506.WXO-DD.bulletins.alphanumeric.20250506.SR.KWAL.20 a file with baseUrl: https://hpfx.collab.science.gc.ca relPath: /20250506/WXO-DD/bulletins/alphanumeric/20250506/SR/KWAL/20/SRWA20_KWAL_062002___50635 sundew_extension: cvt_nws_bulletins-sr:KWAL:SR:3:Direct:20250506200320 id: mzneyRw size: 163
2025-05-06 16:03:22,254 [INFO] 402349 sarracenia.flowcb.log after_accept accepted: (lag: 0.45 ) exchange: xpublic subtopic: 20250506.WXO-DD.bulletins.alphanumeric.20250506.SR.KWAL.20 a file with baseUrl: https://hpfx.collab.science.gc.ca relPath: /20250506/WXO-DD/bulletins/alphanumeric/20250506/SR/KWAL/20/SRCN40_KWAL_062002___7734 sundew_extension: cvt_nws_bulletins-sr:KWAL:SR:3:Direct:20250506200320 id: rTGcmnm size: 147
2025-05-06 16:03:22,254 [INFO] 402349 sarracenia.flowcb.log after_accept accepted: (lag: 0.53 ) exchange: xpublic subtopic: 20250506.WXO-DD.bulletins.alphanumeric.20250506.SR.KWAL.20 a file with baseUrl: https://hpfx.collab.science.gc.ca relPath: /20250506/WXO-DD/bulletins/alphanumeric/20250506/SR/KWAL/20/SRCN40_KWAL_062002___26444 sundew_extension: cvt_nws_bulletins-sr:KWAL:SR:3:Direct:20250506200319 id: kz9EKHa size: 129
2025-05-06 16:03:22,254 [INFO] 402349 sarracenia.flowcb.log after_accept accepted: (lag: 0.53 ) exchange: xpublic subtopic: 20250506.WXO-DD.bulletins.alphanumeric.20250506.SX.KWAL.20 a file with baseUrl: https://hpfx.collab.science.gc.ca relPath: /20250506/WXO-DD/bulletins/alphanumeric/20250506/SX/KWAL/20/SXAK50_KWAL_062002___5577 sundew_extension: cvt_nws_bulletins-sr:KWAL:SX:3:Direct:20250506200319 id: Jf/H/VF size: 174
2025-05-06 16:03:22,733 [INFO] 402349 sarracenia.flowcb.log after_work rejected: 406 mtime not newer /tmp/dual_amis/SRCN40_KWAL_062002___38624
2025-05-06 16:03:22,733 [INFO] 402349 sarracenia.flowcb.log after_work rejected: 406 mtime not newer /tmp/dual_amis/SRCN40_KWAL_062002___46610
2025-05-06 16:03:22,733 [INFO] 402349 sarracenia.flowcb.log after_work rejected: 406 mtime not newer /tmp/dual_amis/SXCN40_KWAL_062002___39285
2025-05-06 16:03:22,733 [INFO] 402349 sarracenia.flowcb.log after_work rejected: 406 mtime not newer /tmp/dual_amis/SRCN40_KWAL_062002___7734
2025-05-06 16:03:22,733 [INFO] 402349 sarracenia.flowcb.log after_work rejected: 406 mtime not newer /tmp/dual_amis/SXAK50_KWAL_062002___5577
2025-05-06 16:03:22,733 [INFO] 402349 sarracenia.flowcb.log after_work downloaded ok: /tmp/dual_amis/SXCN40_KWAL_062002___39285
2025-05-06 16:03:22,733 [INFO] 402349 sarracenia.flowcb.log after_work downloaded ok: /tmp/dual_amis/SXAK50_KWAL_062002___5577
2025-05-06 16:03:22,733 [INFO] 402349 sarracenia.flowcb.log after_work downloaded ok: /tmp/dual_amis/SRCN40_KWAL_062002___7734
2025-05-06 16:03:22,733 [INFO] 402349 sarracenia.flowcb.log after_work downloaded ok: /tmp/dual_amis/WSJP31_RJTD_062002___18004
2025-05-06 16:03:22,733 [INFO] 402349 sarracenia.flowcb.log after_work downloaded ok: /tmp/dual_amis/SRWA20_KWAL_062002___50635
2025-05-06 16:03:22,733 [INFO] 402349 sarracenia.flowcb.log after_work downloaded ok: /tmp/dual_amis/SRCN40_KWAL_062002___26444
2025-05-06 16:03:22,733 [INFO] 402349 sarracenia.flow run 19 messages processed > messageCountMax 10
2025-05-06 16:03:22,734 [INFO] 402349 sarracenia.flow please_stop asked to stop
2025-05-06 16:03:22,734 [INFO] 402349 sarracenia.moth please_stop asked to stop
2025-05-06 16:03:22,734 [INFO] 402349 sarracenia.moth please_stop asked to stop
2025-05-06 16:03:22,734 [INFO] 402349 sarracenia.flow _runHousekeeping on_housekeeping pid: 402349 subscribe/dual_amis instance: 0
2025-05-06 16:03:22,734 [INFO] 402349 sarracenia.flowcb.gather.message on_housekeeping from amqps://anonymous@dd.weather.gc.ca/ messages: good: 7 bad: 0 bytes: 912Bytes average: 130Bytes
2025-05-06 16:03:22,734 [INFO] 402349 sarracenia.flowcb.gather.message on_housekeeping from amqps://anonymous@hpfx.collab.science.gc.ca/ messages: good: 11 bad: 0 bytes: 1.5KiB average: 138Bytes
2025-05-06 16:03:22,734 [INFO] 402349 sarracenia.flowcb.housekeeping.resources on_housekeeping Current cpu_times: user=0.42 system=0.09
2025-05-06 16:03:22,734 [INFO] 402349 sarracenia.flowcb.housekeeping.resources on_housekeeping Current mem usage: 125.2MiB, accumulating count (11 or 18/100 so far) before self-setting threshold
2025-05-06 16:03:22,734 [INFO] 402349 sarracenia.flowcb.log stats version: 3.00.59rc1, started: 2 seconds ago, last_housekeeping: 2.3 seconds ago
2025-05-06 16:03:22,734 [INFO] 402349 sarracenia.flowcb.log stats messages received: 25, accepted: 18, rejected: 7 rate accepted: 72.0% or 7.7 m/s
2025-05-06 16:03:22,734 [INFO] 402349 sarracenia.flowcb.log stats files transferred: 11 bytes: 2.2KiB rate: 980Bytes/sec
2025-05-06 16:03:22,734 [INFO] 402349 sarracenia.flowcb.log stats lag: average: 0.83, maximum: 1.91
2025-05-06 16:03:22,734 [INFO] 402349 sarracenia.flow run 19 messages processed > messageCountMax 10
2025-05-06 16:03:22,734 [INFO] 402349 sarracenia.flow please_stop asked to stop
2025-05-06 16:03:22,735 [INFO] 402349 sarracenia.moth please_stop asked to stop
2025-05-06 16:03:22,735 [INFO] 402349 sarracenia.moth please_stop asked to stop
2025-05-06 16:03:22,735 [INFO] 402349 sarracenia.flow _runHousekeeping on_housekeeping pid: 402349 subscribe/dual_amis instance: 0
2025-05-06 16:03:22,735 [INFO] 402349 sarracenia.flowcb.gather.message on_housekeeping from amqps://anonymous@dd.weather.gc.ca/ messages: good: 0 bad: 0 bytes: 0Bytes average: 0Bytes
2025-05-06 16:03:22,735 [INFO] 402349 sarracenia.flowcb.gather.message on_housekeeping from amqps://anonymous@hpfx.collab.science.gc.ca/ messages: good: 0 bad: 0 bytes: 0Bytes average: 0Bytes
2025-05-06 16:03:22,735 [INFO] 402349 sarracenia.flowcb.housekeeping.resources on_housekeeping Current cpu_times: user=0.42 system=0.09
2025-05-06 16:03:22,735 [INFO] 402349 sarracenia.flowcb.housekeeping.resources on_housekeeping Current mem usage: 125.2MiB, accumulating count (11 or 18/100 so far) before self-setting threshold
2025-05-06 16:03:22,735 [INFO] 402349 sarracenia.flowcb.log stats version: 3.00.59rc1, started: 2 seconds ago, last_housekeeping: 0.0 seconds ago
2025-05-06 16:03:22,735 [INFO] 402349 sarracenia.flowcb.log stats messages received: 0, accepted: 0, rejected: 0 rate accepted: 0.0% or 0.0 m/s
2025-05-06 16:03:22,735 [INFO] 402349 sarracenia.flowcb.log stats files transferred: 0 bytes: 0Bytes rate: 0Bytes/sec
2025-05-06 16:03:22,776 [INFO] 402349 sarracenia.flow close flow/close completed cleanly pid: 402349 subscribe/dual_amis instance: 0
As you can see, it downloaded five files to /tmp/amis. The foreground action is intended to help with debugging, rather than real operations.
[7]:
!sr3 status
2025-05-06 16:03:56,913 402441 [WARNING] sarracenia.config parse_line ./dual_amis.conf:20 broker needs to appear before *subtopic* unless you need different queues to have different settings
2025-05-06 16:03:56,913 402441 [INFO] sarracenia.config finalize subscribe/dual_amis.conf overriding batch for consistency with messageCountMax: 10
status:
Component/Config Processes Rates
State Run Retry Que Lag Last %rej messages Data
----- --- ----- --- --- ---- ---- -------- ----
subscribe/dual_amis stop 0/0 0 0 0.00s n/a 0.0% 0m/s 0B/s
Total Running Configs: 0 ( Processes: 0 missing: 0 stray: 0 )
Memory: uss:0Bytes rss:0Bytes vms:0Bytes
CPU Time: User:0.00s System:0.00s
Pub/Sub Received: 0m/s (0B/s), Sent: 0m/s (0B/s) Queued: 0 Retry: 0, Mean lag: 0.00s
Data Received: 0f/s (0B/s), Sent: 0f/s (0B/s)
Above, you can see there is 1 configuration in your list. You can have hundreds. The columns on the right refer to how many instances you have for each configuration. In the example above, instances is set to 5, so one would expect to see 5 running instances when it would be running. You can start specifc configurations (in this case a subscribe config) with sr3 start subscribe/<config>, or start all active configs from all components (sarra, subscribe, watch, winnow, etc.) with sr3 start
[8]:
!sr3 log subscribe/dual_amis.conf
2025-05-06 16:04:11,004 402457 [WARNING] sarracenia.config parse_line ./dual_amis.conf:20 broker needs to appear before *subtopic* unless you need different queues to have different settings
2025-05-06 16:04:11,005 402457 [INFO] sarracenia.config finalize subscribe/dual_amis.conf overriding batch for consistency with messageCountMax: 10
tail: cannot open '/home/peter/.cache/sr3/log/subscribe_dual_amis_01.log' for reading: No such file or directory
tail: no files remaining
2025-05-06 16:04:11,007 402457 [CRITICAL] root run_command subprocess.run failed err=Command '['tail', '-f', '/home/peter/.cache/sr3/log/subscribe_dual_amis_01.log']' returned non-zero exit status 1.
When running in the background, output needs to go a log file. Since we have only ran this configuration file in the foreground, asking to see the log prints an error about the log being missing. This tells you that the logs are in the ~/.cache/sr3/log directory. Logs can be monitored in real-time with traditional tools such as tail -f or grep.
sr3 stop does what you expect.
Processes can crash. In the sr3 status output above, if the number of processes in the Run column is less than in the Exp (for Expected) one, then it means that some instances have crashed. You can repair it (just start the missing instances) with:
sr3 sanity – start missing instances, also kill strays if any found.
So that’s it, an introduction to running configurations in Sarracenia from the command line.
Conclusion
If all you want to do is obtain data from a data pump in real-time, the easiest way to go is using the command line interface to control some processes that run all the time so that they dump files in a certain directory.
It isn’t very efficient though. When dealing with a large number of files and aiming for high-speed processing, it’s more efficient to have your own application receive notifications about file arrivals rather than scanning a directory. This approach reduces CPU and I/O overhead while improving processing speed.
The easiest way to do that is to add some callbacks to your flows. We’ll cover that next.