API reference

classify_final_dx(topline)

Categorize a string into an overall diagnostic category.

This function uses pattern matching to perform classification. The expected input is the top-line of the final diagnosis.

Examples

julia> classify_final_dx("Parathyroid adenoma")
"BENIGN"

julia> classify_final_dx("Moderate dysplasia")
"INTERMEDIATE"

julia> classify_final_dx("Squamous cell carcinoma")
"MALIGNANT"

julia> classify_final_dx("")

julia> classify_final_dx(missing)

julia> # this outputs `nothing`

classify_final_margin(description, final_topline, final_rest[, overall_status])

Categorize final margin status from final diagnosis fields.

If overall_status is unspecified, it will be computed from classify_final_dx(final_topline).

Examples

julia> classify_final_margin("Jaw removal", "Squamous cell carcinoma", "Margins free of tumour")
"BENIGN"

julia> classify_final_margin("Jaw removal", "Squamous cell carcinoma", "Carcinoma extends to the margin")
"MALIGNANT"

julia> classify_final_margin("Jaw removal", "Gossypiboma", "The entire margin is involved by tumor", "BENIGN")

julia> # no output since diagnosis is benign

final_dataframe(final_table)

Convert the output of final_table() to a DataFrame.

final_table(vec_accession, vec_frozen_text)

Return a two-dimensional vector of case numbers and their associated parsed final diagnosis information.

Example

julia> df = load_raw_data()
[...]

julia> frozens = final_table(df.specnum, df.final_text)
27301-element Vector{Vector}:
[...]

frozen_dataframe(frozen_table)

Convert the output of frozen_table() to a DataFrame.

frozen_table(vec_accession, vec_frozen_text)

Return a two-dimensional vector of case numbers and their associated parsed frozen information.

Example

julia> df = load_raw_data()
[...]

julia> frozens = frozen_table(df.specnum, df.frozen_text)
15615-element Vector{Vector}:
[...]

load_raw_data([path])

Read the first sheet of an XLSX file at path into a DataFrame

Example

julia> df = load_raw_data("path/to/file.xlsx")
6179×12 DataFrame
  Row │ [...]

parse_field_b(str)

Categorize field B of frozen section reports.

Field B should already be designated as one of "BENIGN", "DEFER" or "MALIGNANT", but occasionally other strings may have been entered by the frozen section pathologist.

Examples

julia> parse_field_b("Carcinoma")
"MALIGNANT"

julia> parse_field_b("Negative")
"BENIGN"

julia> parse_field_b("Moderate dysplasia")
"DEFER"

julia> parse_field_b("Weight only")
"GROSS"

julia> parse_field_b("Parathyroid tissue identified")
"OTHER"

julia> parse_field_b(missing)
missing

create_finals_dataframe(data)

Create a DataFrame of parsed final diagnosis data.

This is a utility function to correctly call final_dataframe and final_table in combination.

Example

julia> df = load_raw_data();

julia> create_final_dataframe(df)
27301×8 DataFrame
   Row │ specnum      part    add_part  description                        final_topline                      final_rest  ⋯
       │ Union…       Union…  Union…    Union…                             Union…                             Union…      ⋯
───────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────
     1 │ [...]

create_fs_dataframe(data)

Create a DataFrame of parsed frozen data.

This is a utility function to correctly call frozen_dataframe and frozen_table in combination.

Example

julia> df = load_raw_data();

julia> create_fs_dataframe(df)
15615×11 DataFrame
   Row │ specnum      part    block   add_part  add_block  description                        field_a                     ⋯
       │ Union…       Union…  Union…  Union…    Union…     Union…                             Union…                      ⋯
───────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────
     1 │ [...] 

finals_to_db(db, finals_dataframe)

Write finals_dataframe to SQLite database db as a new table "finals".

fs_to_db(db, fs_dataframe)

Write fs_dataframe to SQLite database db as a new table "fs".

import_block_file!(db, path)

Add rows from an XLSX file with exported block metadata into the block_info table. This table will be created if it does not yet exist.

open_db()
open_db(path)

Create a database connection to path or without argument, data/db.sqlite.

This is an underlying call to SQLite.DB().

transformations_finals_dataframe!(fs_dataframe)

Mutate the final diagnoses DataFrame with defined transformations.

This applies classify_final_dx to generate :overall_status and classify_final_margin to generate :margin_status.

transformations_fs_dataframe!(fs_dataframe)

Mutate the frozens DataFrame with defined transformations.

This applies uppercase_last_word to :fs_path and parse_field_b to generate :interpretation.

import_csv_to_db!(db::DBInterface.Connection, input_csv, table_name)

Create a new table with CSV data

Reads input_csv into db as a new table called table_name. This fails if the table exists.

matches_pattern(patterns, str::AbstractString)

Check if str matches any Regex in patterns.

Examples

julia> using EntFsCorr.Utils: matches_pattern

julia> matches_pattern([r"foo", r"bar", r"baz"], "onefootwo")
true

julia> matches_pattern([r"foo", r"bar", r"baz"], "quuuuux")
false

uppercase_last_word(str)

Return last word of str in uppercase.

Example

julia> using EntFsCorr.Utils: uppercase_last_word

julia> uppercase_last_word("T. Jones")
"JONES"

average_levels_by_overall(
    db::DBInterface.Connection;
    overall_levels_view_name = "overall_levels",
)

Compute average and maximum number of levels per part, stratified by overall status

Example

julia> average_levels_by_overall(db)
6×5 DataFrame
 Row │ overall_interpretation  overall_status  avg_lv_per_block  max_lv_per_block  nrow
     │ String                  String          Float64           Float64           Int64
─────┼───────────────────────────────────────────────────────────────────────────────────
   1 │ BENIGN                  BENIGN                   1.57036           1.5749    5060
   2 │ DEFER                   BENIGN                   1.72268           1.73443    851
   3 │ MALIGNANT               BENIGN                   1.5               1.75         4
   4 │ BENIGN                  MALIGNANT                1.80635           1.82857    315
   5 │ DEFER                   MALIGNANT                1.79213           1.88764    534
   6 │ MALIGNANT               MALIGNANT                1.58543           1.66339   2540

concordance_by_specimen_type(
    db::DBInterface.Connection;
    concordance_table = "concordance_specimen"
)

Compute concordance data by specimen type

Returns a GroupedDataFrame by specimen/part type.

Example

julia> concordance_by_specimen_type(db)
GroupedDataFrame with 10 groups based on key: part_type
First Group (5 rows): part_type = "ent_biopsy"
 Row │ part_type   concordance      nrow   proprow
     │ String      String           Int64  Float64
─────┼────────────────────────────────────────────────
   1 │ ent_biopsy  agree             4141  0.805485
   2 │ ent_biopsy  defer              886  0.17234
   3 │ ent_biopsy  major_undercall     99  0.019257
   4 │ ent_biopsy  major_overcall       9  0.00175063
   5 │ ent_biopsy  minor_undercall      6  0.00116709
⋮
Last Group (2 rows): part_type = "hematolymphoid"
 Row │ part_type       concordance  nrow   proprow
     │ String          String       Int64  Float64
─────┼──────────────────────────────────────────────
   1 │ hematolymphoid  defer            4  0.666667
   2 │ hematolymphoid  agree            2  0.333333

concordance_by_specimen_view!(
    db::DBInterface.Connection[,
    part_type_classifier];
    concordance_table="concordance",
    block_info_table="block_info"
)

Create an SQL view with level of concordance and specimen origin

Also includes staff to see differences in sites.

part_type_classifier is an optional positional argument pointing to a two-column CSV file that maps each part_type to a category.

concordance_by_staff_type(
    db::DBInterface.Connection;
    concordance_table="concordance_staff",
    ingroup=("RP", "RRS", "SXC1", "DXB", "ANB"),
)

Overall levels of agreement stratified by head and neck practice

Uses the view created by concordance_by_staff_view! and computes count of each level of concordance over the group of parts assessed by head and neck pathologists versus those assessed by others.

Returns a named tuple (hn=::DataFrame, non_hn=::DataFrame).

Example

julia> concordance_by_staff_type(db)
(hn = 5×3 DataFrame
 Row │ concordance      nrow   proprow
     │ String           Int64  Float64
─────┼────────────────────────────────────
   1 │ agree             6741  0.828336
   2 │ defer             1125  0.13824
   3 │ major_undercall    209  0.025682
   4 │ major_overcall      54  0.00663554
   5 │ minor_undercall      9  0.00110592, non_hn = 4×3 DataFrame
 Row │ concordance      nrow   proprow
     │ String           Int64  Float64
─────┼────────────────────────────────────
   1 │ agree              796  0.60717
   2 │ defer              476  0.363082
   3 │ major_undercall     31  0.0236461
   4 │ major_overcall       8  0.00610221)

concordance_by_staff_view!(
    db::DBInterface.Connection;
    concordance_table="concordance",
    block_info_table="block_info"m
)

Create an SQL view with level of concordance and FS pathologist

distinct_cases_with_interpretation(db::DBInterface.Connection)

Return number of cases with at least one parsed field_b.

Example

julia> distinct_cases_with_interpretation(db)
6092

interpretation_counts!(db) # view=true

Short-hand for interpretation_counts but creating a new SQLite view.

interpretation_counts(db; view=false)

Return counts of each combination of FS and final calls.

Provides a DataFrame or creates an SQLite view (if view is set to true or function is called with !) of the number of occurrences of each combination of interpretation, margin_status and overall_status.

Example

julia> interpretation_counts(db)
36×4 DataFrame
 Row │ COUNT(*)  interpretation  margin_status  overall_status
     │ Int64     String          String?        String?
─────┼─────────────────────────────────────────────────────────
   1 │     3144  BENIGN          missing        BENIGN
   2 │       14  BENIGN          missing        INTERMEDIATE
   3 │       56  BENIGN          missing        MALIGNANT
   4 │     2222  BENIGN          BENIGN         BENIGN
   5 │        5  BENIGN          BENIGN         INTERMEDIATE
   6 │     2782  BENIGN          BENIGN         MALIGNANT
   7 │        4  BENIGN          INTERMEDIATE   INTERMEDIATE
   8 │      780  BENIGN          MALIGNANT      MALIGNANT
   9 │        2  BENIGN          SUPERSEDED     MALIGNANT
  10 │       96  DEFER           missing        missing
  11 │      666  DEFER           missing        BENIGN
  ⋮  │    ⋮            ⋮               ⋮              ⋮
  27 │        7  MALIGNANT       SUPERSEDED     MALIGNANT
  28 │        2  OTHER           missing        missing
  29 │        6  OTHER           missing        BENIGN
  30 │        4  OTHER           missing        MALIGNANT
  31 │        1  OTHER           BENIGN         BENIGN
  32 │       10  OTHER           BENIGN         MALIGNANT
  33 │        6  OTHER           MALIGNANT      MALIGNANT
  34 │        1  OTHER           OTHER          OTHER
  35 │        1  SUPERSEDED      BENIGN         MALIGNANT
  36 │        1  SUPERSEDED      MALIGNANT      MALIGNANT
                                                15 rows omitted

margin_distance_bins(
    db::DBInterface.Connection;
    margin_distance_table = "margin_distance",
    top_bin = 0.5,
)

Compute discordance rate by margin distance

Bins distances into 0.1 cm increments up to top_bin and reports proportion where concordance is "agree".

Example

julia> margin_distance_bins(db)
6×3 DataFrame
 Row │ distance_lower_bound  concordance_proportion  nrow
     │ Float64               Float64                 Int64
─────┼─────────────────────────────────────────────────────
   1 │                  0.0                0.766667    120
   2 │                  0.1                0.87037     108
   3 │                  0.2                0.904255     94
   4 │                  0.3                0.915493     71
   5 │                  0.4                1.0          24
   6 │                  0.5                0.925373     67

margin_distance_view!(
    db::DBInterface.Connection;
    margin_distance_name = "margin_distance",
    concordance_table_name = "concordance",
    distance_table_name = "combined_fs_after_overrides",
)

Creates a view with lowest distance to margin reported

The view will only include cases called benign on the original frozen and where the overall margin status was either benign or malignant (i.e. no moderate dysplasia at margin).

margin_distance_name is the name of the view to be created.

overall_levels_view!(
    db::DBInterface.Connection;
    # Keyword arguments
    concordance_table_name = "concordance",
    interpretations_table_name = "combined_fs_after_overrides",
    block_table_name = "block_info",
    overall_levels_view_name = "overall_levels",
)

Create view for number of levels cut by overall status

overall_malignancy_risk(
    db::DBInterface.Connection;
    overall_view_name = "overall_fs_vs_final",
    exclude_incomplete = false,
)

Calculate the risk of malignancy by frozen overall interpretation

Uses the view created by overall_view!.

Risk for benign cases may be improperly elevated due to cases where obvious tumor was present but only the margin was sampled. These would be recorded as benign since all frozen tissue was benign. This can be corrected by setting exclude_incomplete = true.

Example

julia> overall_malignancy_risk(db)
3×3 DataFrame
 Row │ overall_interpretation  malignant_proportion  nrow
     │ String                  Float64               Int64
─────┼─────────────────────────────────────────────────────
   1 │ BENIGN                             0.0584958   5385
   2 │ DEFER                              0.371608    1437
   3 │ MALIGNANT                          0.998428    2544

julia> overall_malignancy_risk(db; exclude_incomplete=true)
3×3 DataFrame
 Row │ overall_interpretation  malignant_proportion  nrow
     │ String                  Float64               Int64
─────┼─────────────────────────────────────────────────────
   1 │ BENIGN                             0.0176258   4709
   2 │ DEFER                              0.288805    1063
   3 │ MALIGNANT                          0.999105    1117

overall_view!(
    db::DBInterface.Connection;
    concordance_table_name = "concordance",
    interpretations_table_name = "combined_fs_after_overrides",
    overall_view_name = "overall_fs_vs_final",
)

Create view for overall benign/malignant status

sampling_ratio_by_overall(
    db::DBInterface.Connection;
    overall_view_name = "overall_fs_vs_final",
)

Proportion of frozen blocks for incompletely submitted cases by overall diagnosis.

Uses the view created by overall_view!.

Only cases with a block ratio less than 1 (i.e. not completely frozen) are included. The final diagnostic category must be outright benign or malignant.

Example

julia> result = sampling_ratio_by_overall(db)
6×4 DataFrame
 Row │ overall_interpretation  overall_status  AVG(block_ratio)  COUNT(*)
     │ String                  String          Float64           Int64
─────┼────────────────────────────────────────────────────────────────────
   1 │ BENIGN                  BENIGN                  0.397311       441
   2 │ DEFER                   BENIGN                  0.35526        137
   3 │ MALIGNANT               BENIGN                  0.261538         3
   4 │ BENIGN                  MALIGNANT               0.195013       232
   5 │ DEFER                   MALIGNANT               0.308686       227
   6 │ MALIGNANT               MALIGNANT               0.279124      1424

specimen_by_staff_type(
    db::DBInterface.Connection;
    concordance_table="concordance_specimen",
    ingroup=("RP", "RRS", "SXC1", "DXB", "ANB")
)

Specimen type volume stratified by head and neck practice

Uses the view created by concordance_by_specimen_view! and computes the number of specimens by type assessed by head and neck pathologists versus those assessed by others.

Returns a named tuple (hn=::DataFrame, non_hn=::DataFrame).

Example

julia> specimen_by_staff_type(db)
(hn = 10×3 DataFrame
 Row │ part_type            nrow   proprow
     │ String               Int64  Float64
─────┼─────────────────────────────────────────
   1 │ ent_biopsy            4426  0.543868
   2 │ ent_resection         1445  0.177562
   3 │ endocrine_biopsy       915  0.112435
   4 │ sinonasal_skull        721  0.0885967
   5 │ skin                   326  0.040059
   6 │ neck_dissection        198  0.0243303
   7 │ endocrine_resection     52  0.00638978
   8 │ ent_unclassified        44  0.00540673
   9 │ other                    8  0.000983043
  10 │ hematolymphoid           3  0.000368641, non_hn = 10×3 DataFrame
 Row │ part_type            nrow   proprow
     │ String               Int64  Float64
─────┼────────────────────────────────────────
   1 │ ent_biopsy             713  0.54386
   2 │ ent_resection          254  0.193745
   3 │ skin                   112  0.085431
   4 │ endocrine_biopsy       109  0.0831426
   5 │ sinonasal_skull         53  0.0404272
   6 │ neck_dissection         41  0.0312738
   7 │ other                   12  0.00915332
   8 │ endocrine_resection      9  0.00686499
   9 │ ent_unclassified         5  0.00381388
  10 │ hematolymphoid           3  0.00228833)

main(raw_data, db_file)
main()

Build SQL database from raw data.

Without arguments, this will read from hardcoded file paths and output to data/db.sqlite, otherwise it will read from raw_data and output to db_file.

Example

julia> EntFsCorr.main() # use default paths

julia> EntFsCorr.main("path/to/input.xlsx", "path/to/db.sqlite")

margin_status_from_text(text)

Guess overall status of margins from free text.

This assumes that the overall status of the specimen is malignant. Results for benign or non-neoplastic specimens are not tested.

Example

julia> EntFsCorr.margin_status_from_text("MARGINS FREE OF TUMOR")
"BENIGN"

julia> EntFsCorr.margin_status_from_text("CARCINOMA PRESENT AT THE MEDIAL-LATERAL MARGIN")
"MALIGNANT"

frozens_to_fields([case_number, ]str)

Parse frozen text.

Example

julia> frozen_text = "1AFS: Right anterior margin
           A.  Sufficient for diagnosis
           B.  Malignant
           C.  Squamous cell carcinoma, margin free of tumor
           D.  0.5 cm
           
       2AFS: Right inferior margin
           A. Sufficient for diagnosis
           B. Benign
           C. No tumor present";

julia> EntFsCorr.frozens_to_fields(frozen_text)
2-element Vector{Vector{Union{Nothing, String}}}:
 ["1", "AFS", nothing, nothing, "Right anterior margin", "Sufficient for diagnosis", "Malignant", "Squamous cell carcinoma, margin free of tumor", "0.5 cm", nothing]
 ["2", "AFS", nothing, nothing, "Right inferior margin", "Sufficient for diagnosis", "Benign", "No tumor present", nothing, nothing]

julia> EntFsCorr.frozens_to_fields("ABC25-123", frozen_text)
2-element Vector{Vector{Union{Nothing, String}}}:
 ["ABC25-123", "1", "AFS", nothing, nothing, "Right anterior margin", "Sufficient for diagnosis", "Malignant", "Squamous cell carcinoma, margin free of tumor", "0.5 cm", nothing]
 ["ABC25-123", "2", "AFS", nothing, nothing, "Right inferior margin", "Sufficient for diagnosis", "Benign", "No tumor present", nothing, nothing]

finals_to_fields([case_number, ]str)

Parse final text.

Example

julia> final_text = "Part 1. Thyroid, left, lobectomy (13 grams):
           A.  Follicular adenoma (4.5 cm).
           B.  Nodular thyroid hyperplasia.
           
       Part 2. Central lymph node, biopsy:
           Normocellular parathyroid.";

julia> EntFsCorr.finals_to_fields(final_text)
2-element Vector{Vector{Union{Nothing, String}}}:
 ["1", nothing, "Thyroid, left, lobectomy (13 grams)", "Follicular adenoma (4.5 cm)", "
    B.  Nodular thyroid hyperplasia.
    "]
 ["2", nothing, "Central lymph node, biopsy", "Normocellular parathyroid", nothing]

julia> EntFsCorr.finals_to_fields("ABC25-123", final_text)
2-element Vector{Vector{Union{Nothing, String}}}:
 ["ABC25-123", "1", nothing, "Thyroid, left, lobectomy (13 grams)", "Follicular adenoma (4.5 cm)", "
    B.  Nodular thyroid hyperplasia.
    "]
 ["ABC25-123", "2", nothing, "Central lymph node, biopsy", "Normocellular parathyroid", nothing]

index_block_info_table!(db, block_info_table_name, index_name)

Create an SQLite index on columns specnum, part and block.

The newly created index will be called index_name and the recipient table block_info_table_name.