---
title: "Automating data extraction for SIS"
output:
  html_document:
    toc: true
vignette: >
  %\VignetteIndexEntry{Automating data extraction for SIS}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
```

# Background

After completing a stock assessment report, authors submit key assessment data to the Stock Information System (SIS) for inclusion in StockSmart.

This process has traditionally entailed entering data through a GUI interface. The `stockplotr::extract_sis_data()` function makes this process more efficient and reproducible by automating the extraction of data from model results files and populating templates for SIS submission. Afterwards, the `asar::export_to_sis()` function sends the data to SIS. Below, we explain how this process works.

# Workflow Overview

This process fits within the following workflow:

1. SIS automatically assigns metadata and identifiers to the assessment.

2. Obtain two CSV files containing placeholders for all of the data required by SIS: `sis_assmt_template.csv` (assessment summary data) and `sis_ts_template.csv` (time series data) through one of three ways:

   a. Run `stockplotr::extract_sis_data()` to generate, populate, and export the templates with data originating from a converted model results file. At least one of the following `.rda` files must be present in the `figures/` and/or `tables/` folders:
      * `biomass_figure.rda`
      * `spawning_biomass_figure.rda`
      * `abundance_at_age_figure.rda`
      * `fishing_mortality_figure.rda`
      * `index_figure.rda`
      * `total_catch_table.rda`  
      Ideally, the `key_quantities.csv` file will also be present (in the `key_quantities_dir`).

   b. Generate blank template files by running `asar::create_blank_sis()`.

   c. Locate blank template files in the `report/` folder generated by running `asar::create_template()`.

3. Open the templates and manually fill in any remaining required fields (including the metadata/IDs from Step 1).

4. Run `asar::export_to_sis()` to format and upload this data to a specific Google Drive folder. In the future, this process may occur via an API instead.

5. The SIS administrator will obtain the files and finalize the records.

# How to run `extract_sis_data()`

## Familiarize yourself with the two templates

### sis_assmt_template.csv

This file contains metadata and identifiers for the assessment, as well as summary data for the stock. The columns are as follows:

* **Variable**: SIS database variable name.
* **Explanation**: Variable description. Often includes options.
* **Optional**: Whether the variable is optional or required. If "N", it must be populated before the data can be submitted to SIS.
* **Default**: Default value, if any.
* **Value**: Value of the variable.

### sis_ts_template.csv

This file is formatted to contain time series data for the stock. The columns are as follows:

* **Year**: Observation year.
* **Category**: Time series category. Options include: "Biomass", "Spawners", "Recruitment (age 1)", "Fmort", "Index of Abundance", and "Catch".
* **Primary**: Whether the category is a primary time series category ("Y" or "N"). The following categories can be primary: "Fmort", "Recruitment (age 1)", "Catch", and either "Biomass" OR "Spawners". This function will automatically set the Primary column to "Y" for the applicable time series in each of these categories, and ask the user to confirm which time series should be primary if both "Biomass" and "Spawners" are present. More time series can be added to the template after running `extract_sis_data()`; please see the [Provide additional time series data] section below for more information.
* **Description**: Brief description (e.g., Description for "Fmort" is "F for Fully-Selected Fish").
* **Unit**: Unit of measurement.
* **Value**: Value.

## Export at least one relevant figure/table as an `.rda` file

As the main purpose of this function is to automate the extraction of data into a SIS template, the function requires some data to be present in the `figures/` and/or `tables/` folders. The function will extract data from the `.rda` files created by setting `make_rda = TRUE` in the following plotting functions:

* **Biomass**: `stockplotr::plot_biomass()`
* **Spawning Biomass**: `stockplotr::plot_spawning_biomass()`
* **Recruitment at Age 1**: `stockplotr::plot_abundance_at_age()`
* **Fishing Mortality**: `stockplotr::plot_fishing_mortality()`
* **Index of Abundance**: `stockplotr::plot_index()`
* **Catch**: `stockplotr::table_total_catch()`

## Provide the path to the `key_quantities.csv` file

If you've exported at least one `stockplotr` figure or table, you should have a "key_quantities.csv" file. Data in this file will also be extracted as follows:

### Time series

* The "tot.catch.units" value is used to populate the "Unit" column for the "Catch" time series category.

### Assessment data

These values are extracted and inserted into the key_quantities.csv by running `plot_fishing_mortality()` and `plot_biomass()`:

| key_quantity | Assessment template "Value" |
| :--- | :--- |
| F.end.year | AS_LAST_DATA_YEAR |
| F.MSY.terminal | AS_FMSY |
| B.msy | AS_BMSY |
| B.msy.min | AS_BMSY_MIN |
| B.msy.max | AS_BMSY_MAX |
| B.terminal.year | AS_B_YEAR |
| F.terminal.year | AS_F_YEAR |
| F.target | AS_FTARGET |
| F.limit | AS_FLIMIT |
| B.terminal.est | AS_B_BEST |
| F.terminal.est | AS_F_BEST |
| B.terminal.min | AS_B_MIN |
| B.terminal.max | AS_B_MAX |
| F.terminal.min | AS_F_MIN |
| F.terminal.max | AS_F_MAX |
| F.MSY.terminal.max | AS_FMSY_MAX |
| F.MSY.terminal.min | AS_FMSY_MIN |

* Units for the **Index of Abundance** and **Recruitment (age 1)** time series categories are set to *blank* and "Number of Fish", respectively.

## Provide assessment data before or after running `extract_sis_data()` to populate additional fields

You will probably need to add information to the `sis_assmt_template.csv` to ensure all required information is added. You can do this before or after running `extract_sis_data()`.

A benefit of doing this before running `extract_sis_data()` is that the function will use the information provided to populate additional fields in the `sis_ts_template.csv` file. For example, if you provide the following information in the assessment template:

* "AS_B_UNIT" is used to populate the **Unit** column for the **Biomass** AND **Spawners** time series categories.
* "AS_F_UNIT" is used to populate the **Unit** column for the **Fmort** time series category.

## Provide additional time series data

The `extract_sis_data()` workflow extracts at most one time series for each category. However, you may have multiple time series for a given category. You can add additional time series to the `sis_ts_template.csv` file as per these rules:

* **Fmort, Recruitment, & Catch**: Multiple time series allowed per category, but exactly one must be marked Primary = Y per category.
* **Spawners & Biomass**: Multiple time series allowed across both categories, but only one time series combined across both Spawners and Biomass can be marked Primary = Y.
* **Index**: Multiple time series allowed, but none can be marked Primary = Y (always N).