Key Responsibilities: Automate data quality and reconciliation checks across varied storage layers, including Snowflake, SQL, and RDF/SPARQL databases Test and verify data lineage, governance, and visualization components using Snowflake, data catalogs (ie. Data Hub), Thoughtspot, and other visualization tools Integrate test suites into the core infrastructure orchestrated by Apache Airflow and utilizing Iceberg table formats, while monitoring data pipeline health, alerting, and observability metrics using Prometheus and Grafana Cloud Establish AI Evaluation Loops (Evals) and Guardrails: Build rigorous verification protocols— including structural tests, checks, and watchdog agents—to validate AI-generated artifacts, catch false positives, and ensure all automated outputs are secure, reliable, and free from hallucinations. Integrate automated testing workflows into CI/CD pipelines using Git Hub Actions, ensuring continuous stability and quality gates across all deployment environments Validate ETL and dbt transformations across Data Lakehouses, rigorously testing data progression through a Medallion Architecture Test and automate complex API workflows, validating data payloads across Open API integrations, 3rd party APIs, Graph QL, and AWS APIs (specifically S3)
Must Haves: Data engineering & data testing: dbt, data lakehouse concepts, Medallion architecture Databases & storage testing: SQL, Snowflake, AWS S3, Iceberg Integrating quality check into data pipelines: Apache Airflow API testing & automation: REST/Open API, Graph QL Integrating test automation into CI/CD: Git Hub Actions (or similar like Argo CD/Go CD) Cloud / infrastructure and observability basics: Kubernetes (K8s), Prometheus, Grafana
Nice to Have: Graph databases: RDF / SPARQL Data governance & analytics tools: Data Hub, Thoughtspot AI/ML testing & MLOps: AI evals, guardrails, RAG, vector databases, AI drift monitoring Advanced / emerging data tech: Star Rocks, Duck DB Regulated environments: Gx P, 21 CFR Part 11, HIPAA Clinical / domain-specific data standards: CDISC, ODM, FHIR AI-native tooling: Cursor, Claude Code, Copilot, QA Wolf