--- title: "Scallop End-to-End Workflow" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Scallop End-to-End Workflow} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} editor_options: markdown: wrap: 72 --- ## Overview This vignette walks through the core FishSET workflow on the bundled scallop example data. It starts with raw data loading, moves through QA/QC and data reshaping, and ends with model fitting and diagnostics. ### Setup and data loading We first load FishSET and the example datasets, then place the project in a temporary working folder so the vignette can run without user-specific paths. This keeps the example reproducible while still using the same database-backed workflow as a real project. ```{r setup, include=FALSE} library(FishSET) folderpath <- tempdir() project <- "scallop_vignette" data("scallop") # main data data("scallop_ports") # port data data("tenMNSQR") # spatial data ``` The example data cover three pieces of the workflow: trip-level records, port locations, and the zone polygons used to assign fishing choices. A quick look at their structure helps confirm the variables needed later in the workflow are present. ```{r} str(scallop[, c("TRIPID", "DATE_TRIP", "PERMIT.y", "DDLAT", "DDLON", "ZoneID", "LANDED_OBSCURED", "DOLLAR_OBSCURED")]) str(scallop_ports) str(tenMNSQR, max.level = 1) ``` Now we load the raw data into FishSET using a single project name. The functions save each table to the project database and make the working table available in the R session for the next steps. ```{r} load_maindata(dat = scallop, project = project, over_write = TRUE) load_port(dat = scallop_ports, port_name = "port_name", project = project) load_spatial(spat = tenMNSQR, project = project, name = "tenMNSQR") ``` ## Quality assurance and quality control We begin QA/QC by checking for missing values in the columns that matter for modeling. Removing those rows up front ensures the later formatting and model-fitting steps do not fail on incomplete observations. ```{r} scallop_clean <- na_filter( dat = "scallop_vignetteMainDataTable", project = project, x = c("previous_port_lon", "previous_port_lat", "ZoneID"), remove = TRUE, over_write = TRUE ) ``` Next we run the spatial checks to identify observations that fall on land or outside the intended zone domain. This also parses the date column so we can summarize spatial issues over time. Note: removing observations on land or outside the zone polygons is easier in the FishSET GUI. ```{r} qaqc_out <- spatial_qaqc( dat = "scallop_vignetteMainDataTable", project = project, spat = "scallop_vignettetenMNSQRSpatTable", lon.dat = "DDLON", lat.dat = "DDLAT", date = "DATE_TRIP" ) qaqc_out$spatial_summary qaqc_out$land_plot # Remove observations on land scallop_spat_clean <- qaqc_out$dataset[which(qaqc_out$dataset$ON_LAND != TRUE), ] # Removing extra variables that are no longer needed helps speed up the modeling process cols_to_drop <- c("YEAR", "ON_LAND", "ON_ZONE_BOUNDARY", "EXPECTED_LOC") scallop_spat_clean <- scallop_spat_clean[, !(names(scallop_spat_clean) %in% cols_to_drop)] # Save updated data to FishSET database load_maindata( scallop_spat_clean, project = project, over_write=TRUE) ``` To see how the cleaned trips are distributed across zones, we summarize the observations against the ten-minute-square grid. The tabular output and static plot give a quick read on zone coverage before we reshape the data for modeling. ```{r} zone_out <- zone_summary( dat = "scallop_vignetteMainDataTable", spat = "scallop_vignettetenMNSQRSpatTable", project = project, zone.dat = "ZoneID", zone.spat = "TEN_ID", output = "tab_plot", plot_type = "static", dat_lon = "DDLON", dat_lat = "DDLAT" ) zone_out$table zone_out$plot ``` FishSET provides several other exploratory functions. Here are a few examples: ```{r} # Vessel count by year vessel_count(scallop_vignetteMainDataTable, project, v_id = "PERMIT.y", date = "DATE_TRIP", period = "year", type = "line", output= "plot") # Scale LANDED_OBSCURED to thousands of pounds scallop_vignetteMainDataTable$landed_thousands <- scallop_vignetteMainDataTable$LANDED_OBSCURED / 1000 # Create Catch per Unit Effort variables cpue_out <- cpue(dat =scallop_vignetteMainDataTable, project, xWeight = "landed_thousands", xTime = "TRIP_LENGTH", name = "cpue") # Average CPUE by year and gear code species_catch(cpue_out, project, species = "cpue", date = "DATE_TRIP", group = "GEARCODE", period = "year", fun = "mean", type = "line", output= "plot") ``` More FishSET functions can be found on the [references page](https://noaa-nwfsc.github.io/FishSET/reference/index.html). ## Prepare and format data The model needs a centroid table so it can measure distances from the observed trip location to each alternative zone. Creating the zonal centroid table also confirms that the spatial data and the trip zones line up correctly. ```{r} zone_centroid <- create_centroid( spat = "scallop_vignettetenMNSQRSpatTable", project = project, spatID = "TEN_ID", type = "zonal centroid", output = "centroid table" ) ``` We then define two alternative-choice sets with different minimum-haul filters. Using two versions lets us compare how the available choice set changes as we tighten the inclusion rules. ```{r} create_alternative_choice( dat = "scallop_vignetteMainDataTable", project = project, alt_name = "alt1", zoneID = "ZoneID", occasion = "lon-lat", occasion_var = c("previous_port_lon", "previous_port_lat"), alt_var = "zonal centroid", min_haul = 5 ) create_alternative_choice( dat = "scallop_vignetteMainDataTable", project = project, alt_name = "alt2", zoneID = "ZoneID", occasion = "lon-lat", occasion_var = c("previous_port_lon", "previous_port_lat"), alt_var = "zonal centroid", min_haul = 200 ) ``` Expected catch matrices translate the recent catch history into moving window averages for model inputs. We create one matrix for each alternative-choice definition so the later model specification can use the matching expectation set. ```{r} create_expectations( dat = "scallop_vignetteMainDataTable", project = project, name = "exp_catch1", alt_name = "alt1", catch = "LANDED_OBSCURED", temp_var = "DATE_TRIP", temporal = "daily", temp_window = 7, day_lag = 1, year_lag = 0, empty_catch = NA, empty_expectation = 1e-4 ) create_expectations( dat = "scallop_vignetteMainDataTable", project = project, name = "exp_catch2", alt_name = "alt2", catch = "LANDED_OBSCURED", temp_var = "DATE_TRIP", temporal = "daily", temp_window = 7, day_lag = 1, year_lag = 0, empty_catch = NA, empty_expectation = 1e-4 ) ``` The formatted data step reshapes the project tables into the long format used by the RTMB model code within `fishset_fit()`. We keep the variables needed for the choice model and distance calculation, then repeat the process for the second alternative-choice set. ```{r} format_model_data( project = project, name = "format_1", alt_name = "alt1", zone_id = "ZoneID", unique_obs_id = "TRIPID", select_vars = c("TRIPID", "DATE_TRIP", "ZoneID", "LANDED_OBSCURED"), expectations = "exp_catch1", distance = TRUE, distance_units = "mi" ) format_model_data( project = project, name = "format_2", alt_name = "alt2", zone_id = "ZoneID", unique_obs_id = "TRIPID", select_vars = c("GEARCODE", "ZoneID", "DOLLAR_OBSCURED", "LANDED_OBSCURED"), expectations = "exp_catch2", distance = TRUE, distance_units = "mi" ) ``` ## Model design and fit The first design is a standard conditional logit with expected catch and distance as fixed effects. The second adds area-specific constants so we can compare a more flexible zonal logit specification. ```{r} fishset_design( formula = chosen ~ exp_catch1 + distance, project = project, model_name = "clogit1", formatted_data_name = "format_1", unique_obs_id = "TRIPID", zone_id = "ZoneID", scale = TRUE ) fishset_design( formula = chosen ~ exp_catch2 + distance + ZoneID, project = project, model_name = "zlogit1", formatted_data_name = "format_2", unique_obs_id = "TRIPID", zone_id = "ZoneID", scale = TRUE ) ``` The model-fitting step can take longer than a vignette render budget, so we keep the fitting code in the document but do not evaluate it here. This still shows the exact commands a user would run locally to estimate and print both model tables. The printed tables include estimated coefficients, standard errors, p-values, log-likelihood, AIC, Pseudo R2, and accuracy. Note that in order to plot the predicted probabilities of fishing on a map the `return_full_prob_mat` input for `fishset_fit()` must be set to `TRUE`. ```{r eval=FALSE} clogit1_fit <- fishset_fit(project = project, model_name = "clogit1") print(clogit1_fit) map_predicted_probs(project, fit_name = "clogit1_fit", spat = "scallop_vignettetenMNSQRSpatTable", zone_spat = "TEN_ID", plot_type = "dynamic", output = "plot") zlogit1_fit <- fishset_fit(project = project, model_name = "zlogit1") print(zlogit1_fit) ``` ## Model diagnostics and validation Once the fits exist, we can test the IIA assumption with the Hausman-McFadden test and then check whether residuals show spatial correlation. These diagnostics help determine whether the final model is well specified. ```{r eval=FALSE} clogit1_iia <- fishset_iia_test(project = project, model_name = "clogit1") print(clogit1_iia) zlogit1_iia <- fishset_iia_test(project = project, model_name = "zlogit1") print(zlogit1_iia) ``` ```{r eval=FALSE} clogit1_resid <- model_resid_corr( project = project, model_name = "clogit1", spat = "scallop_vignettetenMNSQRSpatTable", spat_id = "TEN_ID") print(clogit1_resid) plot(clogit1_resid) zlogit1_resid <- model_resid_corr( project = project, model_name = "zlogit1", spat = "scallop_vignettetenMNSQRSpatTable", spat_id = "TEN_ID" ) print(zlogit1_resid) ``` ## Reproducibility FishSET was designed with the aim of reproducibility. All function calls are logged in a dated file. Log files are stored in the `src` folder. Each log call has a functionID and a list of parameters supplies (args). Some logged functions includes kwargs, optional arguments, an output, or a message. The message section is used to save text output from a function call that users may want to reference later, such as the number of number of rows with missing data. For example, the function call ```{r eval=FALSE} filter_table(dat = 'scallop_vignetteMainDataTable', project = project, x = 'GEARCODE', exp = 'GEARCODE==1') ``` returns the following log entry: ```{r eval=FALSE} { "functionID": "filter_table", "args": [ "scallop_vignetteMainDataTable", "scallop_vignette", "GEARCODE", "GEARCODE=='DREDGE-SCALLOP'" ], "kwargs": [], "output": "", "msg": [ { "dataframe": "scallop_vignetteMainDataTable", "vector": "GEARCODE", "FilterFunction": "GEARCODE=='DREDGE-SCALLOP'" } ] } ``` Log entries are written in JSON. Future version of FishSET will include a function that will read the log files and rerun function calls with current or updated data. Logging is built into FishSET functions. However, it is possible to start a new log file using `log_reset`. New log files are started each day. User-created functions, such as likelihoods, can be saved for future use and logged using `log_func_model`.