Description:
In this role, you will play a key part in designing, developing, and maintaining data solutions using Azure Synapse Analytics. You will leverage your expertise in Spark and SQL Database to transform raw data into valuable insights, driving informed decision-making across the organization.
Responsibilities:
- Design and develop data pipelines using Azure Data Factory (ADF) to extract, transform, and load (ETL) data from various sources like relational databases and cloud storage.
- Utilize Spark within Azure Synapse Analytics to process large datasets efficiently, performing complex transformations and aggregations.
- Design and implement data models within Azure Synapse Analytics leveraging SQL Database for efficient querying and analysis.
- Integrate Azure Synapse with external APIs to retrieve and process data from third-party applications.
- Develop and configure Azure Logic Apps to automate workflows and trigger data pipeline execution based on pre-defined events.
- Manage and optimize data storage using Azure Blob Storage, ensuring efficient data access and cost-effectiveness.
- Collaborate with data analysts and business stakeholders to understand data requirements and translate them into actionable data solutions.
- Implement best practices for data security, governance, and monitoring within the Azure Synapse environment.
Requirements:
- 5-6 years of experience in Azure Data Platforms primarily working with Azure Integration Services.
- Experienced in the Design & Development of Modern Data Platforms (Data Lakehouse and Data warehouse) using Azure Data Factory, Azure Databricks, Azure Functions, Azure Data Lake, Azure Synapse DWH, and Azure Logic App.
- Experienced in designing and implementing Big Data solutions for both Batch Processing and Stream Processing using PySpark, Spark SQL and various other components.
- Worked on making transformations of semi-structured data (XML, HTML, JSON) into structured data.
- Experience in troubleshooting and optimizing problematic SQL queries by Indexing, Partitioning, and other techniques.
- Developed data models for Data warehouse, Dimensions and Facts, views and related database structures.
- Creating Business Intelligence Insights on Azure Synapse Data warehouse.
- Monitor, troubleshoot, and perform root cause analysis and resolve all Production Issues.
- Utilize Agile tools such as Jira and Kanban boards, and user stories to track progress, identify bottlenecks, and optimize workflow efficiency.