OpenSearch Transform Job: The Case of the Silent Failure and the Ghost Key
Debugging OpenSearch Transform jobs can feel like searching for a needle in a haystack, especially when the error messages ar...
Feb 13, 2026
// research infrastructure
Derek Weitzel
Distributed Research Infrastructure
Designing and operating production-grade distributed research platforms across national collaborations including OSG, PATh, NRP, and IRIS-HEP.
Focus areas: reliable data movement, scalable throughput computing, and long-lived scientific services engineered for production use.
// current focus
- Federated Kubernetes across research domains
- Reliable high-throughput data movement
- Production-grade distributed research platforms
// systems
Projects
OSG
A national, distributed computing partnership for data-intensive research.
PATh
Partnership to Advance Throughput Computing, bringing together CHTC and OSG t...
National Research Platform
A community-owned research and education platform for sharing compute, data, ...
IRIS-HEP
An institute advancing software R&D for the HL-LHC era of high energy physics...
GP-ENGINE
Great Plains Extended Network of GPUs for Interactive Experimenters.
GP-STOR
Great Plains Scalable Tiered Object Repository providing distributed, resilie...
SciTokens
Federated, capability-based authorization for distributed scientific computing.
// expertise
Technologies
-
Kubernetes
-
Docker
-
Go
-
Python
-
React
-
Node.js
-
Next.js
-
Git
-
GitHub
-
GitHub Copilot
-
Tailwind CSS
-
CSS3
-
HTML5
-
Vercel
-
Flux
-
Astro
-
Nextcloud
-
Falco
-
Helm
-
OpenSearch
-
Elasticsearch
-
Linux
-
NVIDIA
-
JWT
-
Kubernetes
-
Docker
-
Go
-
Python
-
React
-
Node.js
Show all technologies (24)
-
Next.js
-
Git
-
GitHub
-
GitHub Copilot
-
Tailwind CSS
-
CSS3
-
HTML5
-
Vercel
-
Flux
-
Astro
-
Nextcloud
-
Falco
-
Helm
-
OpenSearch
-
Elasticsearch
-
Linux
-
NVIDIA
-
JWT
// writing
Blog
Profiling the XRootD Monitoring Collector
The XRootD Monitoring Collector (collector) receives file transfer accounting messages from XRootD servers.This transfer info...
Jan 31, 2024
Dashboards for Learning Data Visualizations
Creating dashboards and data visualizations are a favorite past time of mine. Also, I jump at any chance to learn a new tech...
Sep 14, 2022
Improving the Open Science Data Federation's Cache Selection
Optimizing data transfers requires tuning many parameters. High latency between the client and a server can decrease data tr...
Jan 22, 2022
XRootD Client Manager
The validation project for XRootD Monitoring is moving to phase 2, scaletesting. Phase 1 focused on correctness of single se...
Oct 11, 2020
GRACC Transition Visualization
The OSG is in the progress of transitioning from an older ElasticSearch (ES) cluster to a new version. Part of this process ...
Mar 08, 2020