De-ID Studio: Document De-Identification for AI and Analytics Pipelines

De-ID Studio is a batch job (not a web service). Subscribe, pull the image, run once, inspect /output files and audit.json. Exit 0 = all files OK; 1 = some files failed; 2 = config error.

IMAGE AND PATHS

export AWS_REGION=us-east-1 export IMAGE_URI=709825985650.dkr.ecr.us-east-1.amazonaws.com/blue-peak/de-id:1.0.1 export TEST_DIR=$HOME/deid-studio-test mkdir -p $TEST_DIR/input $TEST_DIR/output

STEP 1 - LOGIN AND PULL

aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin 709825985650.dkr.ecr.us-east-1.amazonaws.com docker pull $IMAGE_URI

STEP 2 - CREATE SAMPLE INPUT FILES

cat > $TEST_DIR/input/note.txt <<'EOF' Patient: Jane Sample DOB: 03/15/1985 SSN: 123-45-6789 Email: jane.sample@example.com  Phone: (555) 234-5678 Address: 742 Evergreen Terrace, Springfield, IL 62704 Notes: Dr. Smith reviewed chart. Account ending in 4111111111111111 on file. EOF

cat > $TEST_DIR/input/records.csv <<'EOF' name,email,phone,notes Jane Sample,jane.sample@example.com ,555-234-5678,Patient visit John Example,john.example@example.com ,555-987-6543,Follow-up EOF

STEP 3 - TEST DEFAULT REDACT (all categories)

rm -rf $TEST_DIR/output/* docker run --rm
-v $TEST_DIR/input:/input:ro
-v $TEST_DIR/output:/output
-e INPUT_SOURCE=local -e INPUT_PATH=/input
-e OUTPUT_SINK=local -e OUTPUT_PATH=/output
-e AUDIT_LOG_PATH=/output/audit.json
$IMAGE_URI echo "exit code (expect 0): $?"

VERIFY REDACT: grep '[NAME]' $TEST_DIR/output/note.txt grep '[ADDRESS]' $TEST_DIR/output/note.txt grep 'Dr. Smith reviewed chart' $TEST_DIR/output/note.txt grep '[EMAIL]' $TEST_DIR/output/records.csv cat $TEST_DIR/output/audit.json

STEP 4 - TEST EMAIL-ONLY FILTER

rm -rf $TEST_DIR/output/* docker run --rm
-v $TEST_DIR/input:/input:ro
-v $TEST_DIR/output:/output
-e INPUT_SOURCE=local -e INPUT_PATH=/input
-e OUTPUT_SINK=local -e OUTPUT_PATH=/output
-e AUDIT_LOG_PATH=/output/audit.json
-e IDENTIFIER_CATEGORIES=emails
$IMAGE_URI grep '[EMAIL]' $TEST_DIR/output/note.txt grep 'Jane Sample' $TEST_DIR/output/note.txt

STEP 5 - TEST MASK STRATEGY

echo 'SSN: 123-45-6789' > $TEST_DIR/input/mask.txt rm -rf $TEST_DIR/output/* docker run --rm
-v $TEST_DIR/input:/input:ro
-v $TEST_DIR/output:/output
-e INPUT_SOURCE=local -e INPUT_PATH=/input
-e OUTPUT_SINK=local -e OUTPUT_PATH=/output
-e AUDIT_LOG_PATH=/output/audit.json
-e 'STRATEGY_CONFIG={"gov_ids":"mask","emails":"mask"}'
$IMAGE_URI grep '***-**-6789' $TEST_DIR/output/mask.txt

STEP 6 - TEST PSEUDONYMIZE

echo 'Patient Jane Sample' > $TEST_DIR/input/pseudo.txt export MAPPING_KEY=$(openssl rand -hex 32) rm -rf $TEST_DIR/output/* docker run --rm
-v $TEST_DIR/input:/input:ro
-v $TEST_DIR/output:/output
-e INPUT_SOURCE=local -e INPUT_PATH=/input
-e OUTPUT_SINK=local -e OUTPUT_PATH=/output
-e AUDIT_LOG_PATH=/output/audit.json
-e 'STRATEGY_CONFIG={"names":"pseudonymize"}'
-e MAPPING_OUTPUT_PATH=/output/mapping.bin
-e MAPPING_ENCRYPTION_KEY=$MAPPING_KEY
$IMAGE_URI ls -l $TEST_DIR/output/mapping.bin cat $TEST_DIR/output/pseudo.txt

STEP 7 - TEST CONFIG ERROR (expect exit 2)

docker run --rm $IMAGE_URI; echo "exit code (expect 2): $?"

STEP 8 - ECS / EKS

Use fulfillment templates deploy/templates/ecs-run-task.json and deploy/templates/eks-job.yaml. Set image to $IMAGE_URI, configure S3 buckets, task role s3:GetObject on input prefix and s3:PutObject on output prefix, then RunTask or apply Job.

DOCS: https://loopwell.net/docs/configuration  SUPPORT: contact@loopwell.net