Automating data extraction for SIS

Background

After completing a stock assessment report, authors submit key assessment data to the Stock Information System (SIS) for inclusion in StockSmart.

This process has traditionally entailed entering data through a GUI interface. The stockplotr::extract_sis_data() function makes this process more efficient and reproducible by automating the extraction of data from model results files and populating templates for SIS submission. Afterwards, the asar::export_to_sis() function sends the data to SIS. Below, we explain how this process works.

Workflow Overview

This process fits within the following workflow:

  1. SIS automatically assigns metadata and identifiers to the assessment.

  2. Obtain two CSV files containing placeholders for all of the data required by SIS: sis_assmt_template.csv (assessment summary data) and sis_ts_template.csv (time series data) through one of three ways:

    1. Run stockplotr::extract_sis_data() to generate, populate, and export the templates with data originating from a converted model results file. At least one of the following .rda files must be present in the figures/ and/or tables/ folders:

      • biomass_figure.rda
      • spawning_biomass_figure.rda
      • abundance_at_age_figure.rda
      • fishing_mortality_figure.rda
      • index_figure.rda
      • total_catch_table.rda
        Ideally, the key_quantities.csv file will also be present (in the key_quantities_dir).

    2. Generate blank template files by running asar::create_blank_sis().

    3. Locate blank template files in the report/ folder generated by running asar::create_template().

  3. Open the templates and manually fill in any remaining required fields (including the metadata/IDs from Step 1).

  4. Run asar::export_to_sis() to format and upload this data to a specific Google Drive folder. In the future, this process may occur via an API instead.

  5. The SIS administrator will obtain the files and finalize the records.

How to run extract_sis_data()

Familiarize yourself with the two templates

sis_assmt_template.csv

This file contains metadata and identifiers for the assessment, as well as summary data for the stock. The columns are as follows:

  • Variable: SIS database variable name.
  • Explanation: Variable description. Often includes options.
  • Optional: Whether the variable is optional or required. If “N”, it must be populated before the data can be submitted to SIS.
  • Default: Default value, if any.
  • Value: Value of the variable.

sis_ts_template.csv

This file is formatted to contain time series data for the stock. The columns are as follows:

  • Year: Observation year.
  • Category: Time series category. Options include: “Biomass”, “Spawners”, “Recruitment (age 1)”, “Fmort”, “Index of Abundance”, and “Catch”.
  • Primary: Whether the category is a primary time series category (“Y” or “N”). The following categories can be primary: “Fmort”, “Recruitment (age 1)”, “Catch”, and either “Biomass” OR “Spawners”. This function will automatically set the Primary column to “Y” for the applicable time series in each of these categories, and ask the user to confirm which time series should be primary if both “Biomass” and “Spawners” are present. More time series can be added to the template after running extract_sis_data(); please see the Provide additional time series data section below for more information.
  • Description: Brief description (e.g., Description for “Fmort” is “F for Fully-Selected Fish”).
  • Unit: Unit of measurement.
  • Value: Value.

Export at least one relevant figure/table as an .rda file

As the main purpose of this function is to automate the extraction of data into a SIS template, the function requires some data to be present in the figures/ and/or tables/ folders. The function will extract data from the .rda files created by setting make_rda = TRUE in the following plotting functions:

  • Biomass: stockplotr::plot_biomass()
  • Spawning Biomass: stockplotr::plot_spawning_biomass()
  • Recruitment at Age 1: stockplotr::plot_abundance_at_age()
  • Fishing Mortality: stockplotr::plot_fishing_mortality()
  • Index of Abundance: stockplotr::plot_index()
  • Catch: stockplotr::table_total_catch()

Provide the path to the key_quantities.csv file

If you’ve exported at least one stockplotr figure or table, you should have a “key_quantities.csv” file. Data in this file will also be extracted as follows:

Time series

  • The “tot.catch.units” value is used to populate the “Unit” column for the “Catch” time series category.

Assessment data

These values are extracted and inserted into the key_quantities.csv by running plot_fishing_mortality() and plot_biomass():

key_quantity Assessment template “Value”
F.end.year AS_LAST_DATA_YEAR
F.MSY.terminal AS_FMSY
B.msy AS_BMSY
B.msy.min AS_BMSY_MIN
B.msy.max AS_BMSY_MAX
B.terminal.year AS_B_YEAR
F.terminal.year AS_F_YEAR
F.target AS_FTARGET
F.limit AS_FLIMIT
B.terminal.est AS_B_BEST
F.terminal.est AS_F_BEST
B.terminal.min AS_B_MIN
B.terminal.max AS_B_MAX
F.terminal.min AS_F_MIN
F.terminal.max AS_F_MAX
F.MSY.terminal.max AS_FMSY_MAX
F.MSY.terminal.min AS_FMSY_MIN
  • Units for the Index of Abundance and Recruitment (age 1) time series categories are set to blank and “Number of Fish”, respectively.

Provide assessment data before or after running extract_sis_data() to populate additional fields

You will probably need to add information to the sis_assmt_template.csv to ensure all required information is added. You can do this before or after running extract_sis_data().

A benefit of doing this before running extract_sis_data() is that the function will use the information provided to populate additional fields in the sis_ts_template.csv file. For example, if you provide the following information in the assessment template:

  • “AS_B_UNIT” is used to populate the Unit column for the Biomass AND Spawners time series categories.
  • “AS_F_UNIT” is used to populate the Unit column for the Fmort time series category.

Provide additional time series data

The extract_sis_data() workflow extracts at most one time series for each category. However, you may have multiple time series for a given category. You can add additional time series to the sis_ts_template.csv file as per these rules:

  • Fmort, Recruitment, & Catch: Multiple time series allowed per category, but exactly one must be marked Primary = Y per category.
  • Spawners & Biomass: Multiple time series allowed across both categories, but only one time series combined across both Spawners and Biomass can be marked Primary = Y.
  • Index: Multiple time series allowed, but none can be marked Primary = Y (always N).