"Found issues in dataframe returned by bulk validation"
]
},
{
"cell_type": "markdown",
"id": "001dcc6b-0517-4101-8270-686dcde78e55",
"metadata": {},
"source": [
"### Count of resources with issues\n",
"\n",
"Count of resources (unique fullURL) with issues of all severities (even severity \"info\", so maybe no real issue)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "efa56586-2b67-4219-814a-3d679f360faa",
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"len( pd.unique(df['fullUrl']) )"
]
},
{
"cell_type": "markdown",
"id": "7e0fe487-fa30-4c64-8ba2-b13ac20a7714",
"metadata": {},
"source": [
"### Grouped issues with aggregation of codesystems sorted by count of affected resources\n",
"\n",
"Issues grouped by additional aggregation of Codesystems (e.g. ICD10) by removing the different codes of same codesystem resulting in no separate issue for each used code (e.g. ICD10-Code) of the code system\n",
"### Grouped issues with aggregation of codesystems sorted by severty\n",
"\n",
"Issues grouped by additional aggregation of Codesystems (e.g. ICD10) by removing the different codes of same codesystem resulting in no separate issue for each used code (e.g. ICD10-Code) of the code system\n",
"### Grouped issues without aggregation of codesystems\n",
"\n",
"Issues and count of affected resources sorted by amount of affected resources due to no aggregation of codesystem (for additional aggregation of codesystems see upper sections). This will show a separate issue for each used code used from a codesystem"
"#### Show only issues with severity \"error\" grouped by codesystems\n",
"\n",
"Show grouped issues with filter on severity \"error\"\n",
"\n",
"Issues grouped by additional aggregation of Codesystems (e.g. ICD10) by removing the different codes of same codesystem resulting in no separate issue for each used code (e.g. ICD10-Code) of the code system\n",
"#### Grouped issues with severity \"error\" without aggregation of codesystems\n",
"\n",
"Issues and count of affected resources sorted on amount of affected resources\n",
"Since no aggregation of codesystem (for additional aggregation of codesystems see upper sections) this will show a separate issue for each used code used from a codesystem"
"myerror = \"Condition.code.coding:icd10-gm.version: minimum required = 1, but only found 0 (from https://www.medizininformatik-initiative.de/fhir/core/modul-diagnose/StructureDefinition/Diagnose|2024.0.0)\"\n",
"\n",
"# Use Python syntax:\n",
"# df[df['diagnostics']==myerror]\n",
"#\n",
"# or use df.query\n",
"# https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.query.html and https://docs.python.org/3/reference/lexical_analysis.html#f-strings:\n",
"df_query = f'diagnostics==\"{myerror}\"'\n",
"\n",
"df.query(df_query)"
]
},
{
"cell_type": "markdown",
"id": "5f4146e5-ce88-42e0-b1ac-41b93c1d59f1",
"metadata": {},
"source": [
"## Info\n",
"\n",
"Information concerning the dataframe, e.g. dataframe memory usage"
]
},
{
"cell_type": "markdown",
"id": "2be8ee0e-5c48-4453-804e-c2db23115bd9",
"metadata": {},
"source": [
"### Dataframe memory usage"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "73994859-6824-42e3-9ee3-9570ef9183a8",
"metadata": {},
"outputs": [],
"source": [
"df.info()\n",
"df.memory_usage(deep=True)"
]
},
{
"cell_type": "markdown",
"id": "c46b93d4-35ad-427d-bda8-44c42f6b91a1",
"metadata": {},
"source": [
"### Head - Returns first rows of dataframe"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2e521216-ad7d-4ca6-8e04-7d86435a3a6a",
"metadata": {},
"outputs": [],
"source": [
"df.head()"
]
},
{
"cell_type": "markdown",
"id": "6311f067-72d1-4a32-91fe-585ebfb74c55",
"metadata": {},
"source": [
"## Snippets\n",
"\n",
"Additional code snippets"
]
},
{
"cell_type": "markdown",
"id": "ff1ed096-dc18-492b-988b-8c5b7899adb9",
"metadata": {},
"source": [
"### Markdown generation\n",
"\n",
"How to generate table in markdown format (e.g. for CI/CD status report)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8843debf-ca4c-409c-89eb-8eba64432438",
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"# Reserved char pipe | has to be escaped by | (https://github.com/astanin/python-tabulate/issues/241)\n",
"df_escaped = df.applymap(lambda s: s.replace('|','\\\\|') if isinstance(s, str) else s)\n",