EntFsCorr.jl documentation

classify_final_dx(topline)

Categorize a string into an overall diagnostic category.

This function uses pattern matching to perform classification. The expected input is the top-line of the final diagnosis.

Examples

julia> classify_final_dx("Parathyroid adenoma")
"BENIGN"

julia> classify_final_dx("Moderate dysplasia")
"INTERMEDIATE"

julia> classify_final_dx("Squamous cell carcinoma")
"MALIGNANT"

julia> classify_final_dx("")

julia> classify_final_dx(missing)

julia> # this outputs `nothing`

classify_final_margin(description, final_topline, final_rest[, overall_status])

Categorize final margin status from final diagnosis fields.

If overall_status is unspecified, it will be computed from classify_final_dx(final_topline).

Examples

julia> classify_final_margin("Jaw removal", "Squamous cell carcinoma", "Margins free of tumour")
"BENIGN"

julia> classify_final_margin("Jaw removal", "Squamous cell carcinoma", "Carcinoma extends to the margin")
"MALIGNANT"

julia> classify_final_margin("Jaw removal", "Gossypiboma", "The entire margin is involved by tumor", "BENIGN")

julia> # no output since diagnosis is benign

final_dataframe(final_table)

Convert the output of final_table() to a DataFrame.

final_table(vec_accession, vec_frozen_text)

Return a two-dimensional vector of case numbers and their associated parsed final diagnosis information.

Example

julia> df = load_raw_data()
[...]

julia> frozens = final_table(df.specnum, df.final_text)
27301-element Vector{Vector}:
[...]

frozen_dataframe(frozen_table)

Convert the output of frozen_table() to a DataFrame.

frozen_table(vec_accession, vec_frozen_text)

Return a two-dimensional vector of case numbers and their associated parsed frozen information.

Example

julia> df = load_raw_data()
[...]

julia> frozens = frozen_table(df.specnum, df.frozen_text)
15615-element Vector{Vector}:
[...]

load_raw_data([path])

Read the first sheet of an XLSX file at path into a DataFrame

Example

julia> df = load_raw_data("path/to/file.xlsx")
6179×12 DataFrame
  Row │ [...]

parse_field_b(str)

Categorize field B of frozen section reports.

Field B should already be designated as one of "BENIGN", "DEFER" or "MALIGNANT", but occasionally other strings may have been entered by the frozen section pathologist.

Examples

julia> parse_field_b("Carcinoma")
"MALIGNANT"

julia> parse_field_b("Negative")
"BENIGN"

julia> parse_field_b("Moderate dysplasia")
"DEFER"

julia> parse_field_b("Weight only")
"GROSS"

julia> parse_field_b("Parathyroid tissue identified")
"OTHER"

julia> parse_field_b(missing)
missing

create_finals_dataframe(data)

Create a DataFrame of parsed final diagnosis data.

This is a utility function to correctly call final_dataframe and final_table in combination.

Example

julia> df = load_raw_data();

julia> create_final_dataframe(df)
27301×8 DataFrame
   Row │ specnum      part    add_part  description                        final_topline                      final_rest  ⋯
       │ Union…       Union…  Union…    Union…                             Union…                             Union…      ⋯
───────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────
     1 │ [...]

create_fs_dataframe(data)

Create a DataFrame of parsed frozen data.

This is a utility function to correctly call frozen_dataframe and frozen_table in combination.

Example

julia> df = load_raw_data();

julia> create_fs_dataframe(df)
15615×11 DataFrame
   Row │ specnum      part    block   add_part  add_block  description                        field_a                     ⋯
       │ Union…       Union…  Union…  Union…    Union…     Union…                             Union…                      ⋯
───────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────
     1 │ [...] 

finals_to_db(db, finals_dataframe)

Write finals_dataframe to SQLite database db as a new table "finals".

fs_to_db(db, fs_dataframe)

Write fs_dataframe to SQLite database db as a new table "fs".

open_db()

Create a database connection to data/db.sqlite).

This is an underlying call to SQLite.DB().

transformations_finals_dataframe!(fs_dataframe)

Mutate the final diagnoses DataFrame with defined transformations.

This applies classify_final_dx to generate :overall_status and classify_final_margin to generate :margin_status.

transformations_fs_dataframe!(fs_dataframe)

Mutate the frozens DataFrame with defined transformations.

This applies uppercase_last_word to :fs_path and parse_field_b to generate :interpretation.

matches_pattern(patterns, str)

Check if str matches any Regex in patterns.

Examples

julia> using EntFsCorr.Utils: matches_pattern

julia> matches_pattern([r"foo", r"bar", r"baz"], "onefootwo")
true

julia> matches_pattern([r"foo", r"bar", r"baz"], "quuuuux")
false

uppercase_last_word(str)

Return last word of str in uppercase.

Example

julia> using EntFsCorr.Utils: uppercase_last_word

julia> uppercase_last_word("T. Jones")
"JONES"

main()

Build SQL database from raw data.

This will read from hardcoded file paths and output to data/db.sqlite, so will not work without modifications to the underlying functions.

margin_status_from_text(text)

Guess overall status of margins from free text.

This assumes that the overall status of the specimen is malignant. Results for benign or non-neoplastic specimens are not tested.

Example

julia> EntFsCorr.margin_status_from_text("MARGINS FREE OF TUMOR")
"BENIGN"

julia> EntFsCorr.margin_status_from_text("CARCINOMA PRESENT AT THE MEDIAL-LATERAL MARGIN")
"MALIGNANT"

frozens_to_fields([case_number, ]str)

Parse frozen text.

Example

julia> frozen_text = "1AFS: Right anterior margin
           A.  Sufficient for diagnosis
           B.  Malignant
           C.  Squamous cell carcinoma, margin free of tumor
           D.  0.5 cm
           
       2AFS: Right inferior margin
           A. Sufficient for diagnosis
           B. Benign
           C. No tumor present";

julia> EntFsCorr.frozens_to_fields(frozen_text)
2-element Vector{Vector{Union{Nothing, String}}}:
 ["1", "AFS", nothing, nothing, "Right anterior margin", "Sufficient for diagnosis", "Malignant", "Squamous cell carcinoma, margin free of tumor", "0.5 cm", nothing]
 ["2", "AFS", nothing, nothing, "Right inferior margin", "Sufficient for diagnosis", "Benign", "No tumor present", nothing, nothing]

julia> EntFsCorr.frozens_to_fields("ABC25-123", frozen_text)
2-element Vector{Vector{Union{Nothing, String}}}:
 ["ABC25-123", "1", "AFS", nothing, nothing, "Right anterior margin", "Sufficient for diagnosis", "Malignant", "Squamous cell carcinoma, margin free of tumor", "0.5 cm", nothing]
 ["ABC25-123", "2", "AFS", nothing, nothing, "Right inferior margin", "Sufficient for diagnosis", "Benign", "No tumor present", nothing, nothing]

finals_to_fields([case_number, ]str)

Parse final text.

Example

julia> final_text = "Part 1. Thyroid, left, lobectomy (13 grams):
           A.  Follicular adenoma (4.5 cm).
           B.  Nodular thyroid hyperplasia.
           
       Part 2. Central lymph node, biopsy:
           Normocellular parathyroid.";

julia> EntFsCorr.finals_to_fields(final_text)
2-element Vector{Vector{Union{Nothing, String}}}:
 ["1", nothing, "Thyroid, left, lobectomy (13 grams)", "Follicular adenoma (4.5 cm)", "
    B.  Nodular thyroid hyperplasia.
    "]
 ["2", nothing, "Central lymph node, biopsy", "Normocellular parathyroid", nothing]

julia> EntFsCorr.finals_to_fields("ABC25-123", final_text)
2-element Vector{Vector{Union{Nothing, String}}}:
 ["ABC25-123", "1", nothing, "Thyroid, left, lobectomy (13 grams)", "Follicular adenoma (4.5 cm)", "
    B.  Nodular thyroid hyperplasia.
    "]
 ["ABC25-123", "2", nothing, "Central lymph node, biopsy", "Normocellular parathyroid", nothing]