Dagster Developer(Data & ML pipeline Orchestration Engineer)
Job Description:
|
Job Role |
|
|
Position: |
Dagster Developer (Data & ML Pipeline Orchestration Engineer) |
|
Summary |
|
The Dagster Developer will be responsible for designing, optimizing, and maintaining orchestration workflows for AI/ML pipelines using Dagster on AWS. The role requires strong experience in workflow automation, monitoring, and performance optimization. |
|
Key Responsibilities |
|
• Overall Responsibilities •Develop, deploy, and optimize Dagster-based orchestration pipelines •Monitor and troubleshoot infrastructure-related issues •Collaborate with MLOps engineers to enhance infrastructure support for AI/ML workloads •Ensure high availability and disaster recovery strategies are in place Technical Responsibilities •Integrate Dagster with AWS services like S3, Lambda, Step Functions, Glue, SageMaker, Redshift, and Athena. •Ensure data pipeline reliability, fault tolerance, and scalability. •Design event-driven workflows using AWS EventBridge, SNS, and SQS. •Implement observability and monitoring for pipeline execution and failures. •Automate data quality checks and lineage tracking. •Collaborate with MLOps engineers to orchestrate feature engineering, model training, and deployment pipelines. •Optimize execution time and parallel processing for ML workloads •Collaborate with MLOps and DevOps teams to enhance deployment pipelines. •Experience with DevOps practices, CI/CD pipelines, and GitHub/GitLab workflows. •Strong team player with ability to collaborate effectively across different teams |
Key Skills :
Company Profile
Client an ISO9001 & ISO27001 company. We are Microsoft Gold certified partner also one among the leading Information Technology (IT) Consulting firms of the present generation. We help Clients in achieving the best possible results, efficiency, and performance in their Business and Operations from all their investments in IT. Our services include Automation Testing, DevOps, Business Intelligence, Technology Consulting, Quality Assurance, and Creative services with end-to-end consulting capabilities spanning across key industry domains. Client operates from its strategically located offices in Noida, USA and Canada to ensure highest availability of our resources to the Clients on a 24x7 basis. Our onsite-offshore model for service deployment and delivery ensures greater Client engagement levels and increased Client Satisfaction.
Apply Now
- Interested candidates are requested to apply for this job.
- Recruiters will evaluate your candidature and will get in touch with you.