Bulk-FHIR-Validation/home/fhir-validation.ipynb

Output options for rendering the dataframes/tables

In [ ]:

import pandas as pd
# how many rows of table to show
pd.set_option("display.max_rows", 1000)
# we want to see the full error messages (often longer than default colwidth)
pd.set_option("max_colwidth", 10000)

Validate and render validation results of one FHIR-Resource

In [ ]:

#from fhirvalidation import Validator
#validator = Validator()
#validator.validate_resource_and_render_validation_outcome ('Condition/resource1')

Bulk validation of found resources by FHIR Search

Validate all resources from FHIR search results

Validate conditions

In [ ]:

from fhirvalidation import Validator

validator = Validator()

# Set auth parameters for your FHIR server (if you do not want to use basic auth credentials from the environment variables in .env)
# Documentation: https://requests.readthedocs.io/en/latest/user/authentication/#basic-authentication
# validator.requests_kwargs['auth'] = ( 'myusername', 'mypassword )

# Search for all resources of the resource_type
search_parameters = {}

# Search resources with for certain code
#search_parameters={"code": "A00.0"}

df = validator.search_and_validate(resource_type="Condition", search_parameters=search_parameters, limit=10000)

Issues

Found issues in dataframe returned by bulk validation

Count of resources with issues

Count of resources (unique fullURL) with issues of all severities (even severity "info", so maybe no real issue)

In [ ]:

import pandas as pd
len( pd.unique(df['fullUrl']) )

Grouped issues with aggregation of codesystems sorted by count of affected resources

Issues grouped by additional aggregation of Codesystems (e.g. ICD10) by removing the different codes of same codesystem resulting in no separate issue for each used code (e.g. ICD10-Code) of the code system

Sorted by count of affected resources

In [ ]:

df[['severity', 'location_aggregated', 'diagnostics_aggregated', 'fullUrl']].groupby(["severity", "location_aggregated", "diagnostics_aggregated"]).count().sort_values(['fullUrl'], ascending=False)

Grouped issues with aggregation of codesystems sorted by severty

Issues grouped by additional aggregation of Codesystems (e.g. ICD10) by removing the different codes of same codesystem resulting in no separate issue for each used code (e.g. ICD10-Code) of the code system

Sorterd by severity

In [ ]:

df[['severity', 'location_aggregated', 'diagnostics_aggregated', 'fullUrl']].groupby(["severity", "location_aggregated", "diagnostics_aggregated"]).count().sort_values(['severity','fullUrl'], ascending=False)

Grouped issues without aggregation of codesystems

Issues and count of affected resources sorted by amount of affected resources due to no aggregation of codesystem (for additional aggregation of codesystems see upper sections). This will show a separate issue for each used code used from a codesystem

In [ ]:

df[['severity', 'location', 'diagnostics', 'fullUrl']].groupby(["severity", "location", "diagnostics"]).count().sort_values(['fullUrl'], ascending=False)

Filter on severity "error"

Show only issues filtered by severity "error"

Count of resources with severity "error"

In [ ]:

len( pd.unique(df[df['severity']=="error"]['fullUrl']) )

Show only issues with severity "error" grouped by codesystems

Show grouped issues with filter on severity "error"

Issues grouped by additional aggregation of Codesystems (e.g. ICD10) by removing the different codes of same codesystem resulting in no separate issue for each used code (e.g. ICD10-Code) of the code system

Sorted by count of affected resources

In [ ]:

df.query('severity=="error"')[['location_aggregated', 'diagnostics_aggregated', 'fullUrl']].groupby(["location_aggregated", "diagnostics_aggregated"]).count().sort_values(['fullUrl'], ascending=False)

Grouped issues with severity "error" without aggregation of codesystems

Issues and count of affected resources sorted on amount of affected resources Since no aggregation of codesystem (for additional aggregation of codesystems see upper sections) this will show a separate issue for each used code used from a codesystem

In [ ]:

df.query('severity=="error"')[['location', 'diagnostics', 'fullUrl']].groupby(["location", "diagnostics"]).count().sort_values(['fullUrl'], ascending=False)

Resources with a specific error

In [ ]:

myerror = "Condition.code.coding:icd10-gm.version: minimum required = 1, but only found 0 (from https://www.medizininformatik-initiative.de/fhir/core/modul-diagnose/StructureDefinition/Diagnose|2024.0.0)"

# Use Python syntax:
# df[df['diagnostics']==myerror]
#
# or use df.query
# https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.query.html and https://docs.python.org/3/reference/lexical_analysis.html#f-strings:
df_query = f'diagnostics=="{myerror}"'

df.query(df_query)

Info

Information concerning the dataframe, e.g. dataframe memory usage

Dataframe memory usage

In [ ]:

df.info()
df.memory_usage(deep=True)

Head - Returns first rows of dataframe

In [ ]:

df.head()

Snippets

Additional code snippets

Markdown generation

How to generate table in markdown format (e.g. for CI/CD status report)

In [ ]:

# Reserved char pipe | has to be escaped by | (https://github.com/astanin/python-tabulate/issues/241)
df_escaped = df.applymap(lambda s: s.replace('|','\\|') if isinstance(s, str) else s)

print(df_escaped[['severity', 'location_aggregated', 'diagnostics_aggregated', 'fullUrl']].groupby(["severity", "location_aggregated", "diagnostics_aggregated"]).count().sort_values(['fullUrl'], ascending=False).to_markdown(tablefmt="github") )

Navigate validation results dataframe with interactive user interface

Use interactive UI to navigate and filter the dataframe

Documentation: English / German

In [ ]:

# Install pip package in the current Jupyter kernel
import sys
!{sys.executable} -m pip install pygwalker

In [ ]:

# render dataframe with pygwalker
import pygwalker as pyg
walker = pyg.walk(df)

In [ ]: