Data Engineer

Description:

Virtuosoft is seeking a skilled and experienced Data Engineer to join their dynamic team. The ideal candidate will have a strong background in building scalable data pipelines, working with real-time streaming solutions, and managing cloud-based infrastructure. The role involves designing and maintaining data pipelines, working with Apache Kafka, and utilizing AWS and Oracle Cloud Infrastructure to build and optimize data architecture. This is an excellent opportunity for someone passionate about data architecture and working in a collaborative environment.

Key Responsibilities:

• Design, build, and maintain robust data pipelines using Python, SQL, and Apache Airflow.

• Implement real-time streaming data solutions with Apache Kafka.

• Develop and manage infrastructure-as-code with Terraform; orchestrate deployments via GitOps workflows.

• Manage containerized workloads on Kubernetes, ensuring scalability and availability.

• Build and support data infrastructure using AWS and Oracle Cloud Infrastructure (OCI).

• Collaborate with stakeholders to design efficient data models and optimize architecture.

• Monitor, troubleshoot, and enhance ETL/ELT workflows.

• Integrate with visualization tools like Power BI and Apache Superset.

• Ensure data security, quality, and compliance across systems.

• Contribute to documentation, code reviews, and team best practices.

Required Skills & Qualifications:

• Proficiency in Python for data processing and automation.

• Strong SQL skills and data warehousing knowledge.

• Hands-on experience with Apache Airflow and Kafka.

• Experience managing applications in Kubernetes environments.

• Solid understanding of AWS (S3, Redshift, Lambda, ECS) and OCI services.

• Working knowledge of Terraform and GitOps CI/CD workflows.

• Exposure to BI tools like Power BI and Apache Superset.

• Familiarity with Agile and DevOps practices.

• Strong analytical, problem-solving, and communication skills.

Preferred Qualifications:

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.

• Certifications in AWS, OCI, or Kubernetes.

• Experience with tools like Spark or dbt.

• Understanding of data governance and security practices.