MeDaX Pipeline

📋 Description

The MeDaX pipeline transforms healthcare data from FHIR databases into Neo4j graph databases. This conversion enables efficient searching, querying, and analyses of interconnected health data that would otherwise be complex to retrieve using traditional SQL databases.

✨ Features

• Seamless conversion from FHIR to Neo4j graph structure
• Support for patient-centric data retrieval using FHIR's $everything operation
• Configurable batch processing for handling large datasets
• Docker-based deployment for easy setup and portability
• Compatible with public FHIR servers (e.g., HAPI FHIR) and private authenticated instances

⚙️ Prerequisites

• Docker with the Docker Compose plugin
• A FHIR database with API access and the $everything operation enabled for retrieving patient data
  • Alternatively: Use a public FHIR server such as HAPI FHIR (default configuration)
• Optional: Semspect license
  • free scientific licenses can be requested
  • helps with graph exploration

🚀 Installation

Setup

1. Clone this repository

2. Create an environment configuration file

3. Configure the environment variables in .env:

   • For HAPI test server (default): No changes needed
   • For custom FHIR server:
     • Change MODE to anything else
     • Uncomment and set URL, PASSWORD, and USERNAME variables
     • Adjust BATCH_SIZE and NUMBER_OF_PATIENTS according to your needs
     • Configure any required proxy settings

4. If needed, modify proxy settings in the Dockerfile

   • Uncomment and set proxy variables

Running the Pipeline

Option A: Connect to an existing FHIR server (e.g. HAPI)

Start the containers:

docker compose up --build

Stop and clean up (between runs):

docker compose down --volumes

Complete removal (containers and images):

docker compose down --volumes --rmi all

Option B: Build a local FHIR server with POLAR data

This spins up a local Blaze FHIR server pre-loaded with synthetic POLAR test bundles, using the blaze Compose profile.

Setup: in your .env, set:

COMPOSE_PROFILES=blaze
FHIR_SERVER_URL=http://blaze:8080/fhir

Start the containers:

docker compose up --build

Stop and clean up (between runs):

docker compose down --volumes

Complete removal (containers and images):

docker compose down --volumes --rmi all

Option C: Build a local FHIR server with POLAR data and a local keycloak server

This builds on Option B and adds a local Keycloak instance plus an OAuth2 proxy in front of Neo4j, using the blaze, keycloak, and proxy Compose profiles.

Setup: in your .env, set:

COMPOSE_PROFILES=blaze,keycloak,proxy
FHIR_SERVER_URL=http://blaze:8080/fhir

Start the containers:

Init containers

docker compose up --build

Create realm, client and user

1. Open keycloak in a browser: localhost:4040 and login with user: admin and password: admin
2. Create the realm "testPipeline"
3. Create a client:

• Client ID: neo4j-client
• Always display in UI "on"
• Hit the "Next" button
• Client authentification: on
• Hit the "Next" button
• Root URL: http://localhost:8082
• Home URL: http://localhost:8082
• Valid redirects URIs: http://localhost:8082/oauth2/callback
• Valid post logout redirect URIs: http://localhost:8082/*
• Web origin: http://localhost:8082
• Hit the "Save" button
• Go to the "Credentials" tab
• Copy the "Client Secret" to "NEO4J_OAUTH_SECRET" in your .env

4. Create a user:

• "Email verified": on
• Choose a user name
• Use random mail, e.g.: test@example.com
• Hit the "Create button"
• Click on the new user, go to the "Credentials" tab
• Set a password and disable "Temporary"

Restart containers without destroying the volumes

docker compose down
docker compose up

Stop

docker compose down

Note: When adding --volumes to down, the keycloak setup will be lost

Complete removal (containers and images):

docker compose down --volumes --rmi all

Note: Depending on your Docker installation, you might need to use docker-compose instead of docker compose. Note: With COMPOSE_PROFILES set in your .env, every docker compose command automatically includes the right services - no -f files needed.

🔍 Accessing the Neo4j Database

Once the pipeline has completed processing, you can access the Neo4j database:

1. Open your browser and navigate to http://localhost:8080/ or http://localhost:8082/ if you used keycloak (Option C, served via the proxy profile's neo4j-proxy service)
2. Connect by clicking the button, user name and password are not needed (disabled by config)

📊 Example Queries

Here are some basic Cypher queries to get you started with exploring your health data:

// Count all nodes by type
MATCH (n) RETURN labels(n) as NodeType, count(*) as Count;

// Find all records for a specific patient
MATCH (p:Patient {id: 'patient-id'})-[r]-(connected)
RETURN p, r, connected;

// Retrieve all medication prescriptions
MATCH (m:Medication)-[r]-(p:Patient)
RETURN m, r, p;

❓ Troubleshooting

Common Issues:

• Connection refused to FHIR server: Check your network settings and ensure the FHIR server is accessible from within the Docker container.
• Authentication failures: Verify your credentials in the .env file.
• Container startup failures: Ensure all required Docker ports are available and not used by other applications.
• No data found in fhir bundle: Ensure that the FHIR server is up and responding patient data. Try set the COMPLEX_PATIENTS variable to FALSE in your .env file. Some FHIR servers might not support the FHIR search logic.

📚 Architecture

The MeDaX pipeline consists of the following components:

1. FHIR Client: Connects to the FHIR server and retrieves patient data
2. Data Transformer: Converts FHIR resources into graph entities and relationships
3. Reference Processor: Converts references to relationships
4. BioCypher Adapter: Prepares the transformed data for Neo4j admin import
5. Neo4j Database: Stores and serves the graph representation of the health data

✍️ Citation

If you use the MeDaX pipeline in your research, please cite: 10.5281/zenodo.15229077 and Mazein, I and Gebhardt, T et al. MeDaX, a knowledge graph on FHIR.

🙏 Acknowledgements

• We are leveraging BioCypher to create the Neo4j admin input.
  • Remark: We introduced slight adjustments to BioCypher's code to support batching.
• We used BioCypher's git template as a starting point for our development:
  • Lobentanzer, S., BioCypher Consortium, & Saez-Rodriguez, J. Democratizing knowledge representation with BioCypher [Computer software]. https://github.com/biocypher/biocypher
• We used synthetic data generated with Synthea during the development process. This data is provided in the testData folder.
• We are using the HAPI R4 server while developing and showcasing the capabilities of our tool.
• This project has been funded by the BMBF, FKZ: 01ZZ2019.