Data Engineer - Azure (Remote)

Technologies & Languages

Azure

Databricks

SQL Sever

ADF

Snowflake

Data Cleaning

ETL

Azure Devops

Intermediate Python/Pyspark

Intermediate SQL

Beginners' knowledge/willingness to learn Spotfire

Data Ingestion

Familiarity with CI/CD or Agile

Must have:

Azure – VM, Data Lake, Data Bricks, Data Factory, Azure DevOps

Python/Spark (PySpark)

SQL

Good to have:

Docker

Kubernetes

Scala

He/she should have a good understanding in:

How to build pipelines – ETL and Injection

Data Warehousing

Monitoring

Responsibilities:

Must be able to write quality code and build secure, highly available systems.

Assemble large, complex data sets that meet functional / non-functional business requirements.

Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc with the guidance.

Create data tools for analytics and data scientist team members that assist them in building and optimizing our product into an innovative industry leader.

Monitoring performance and advising any necessary infrastructure changes.

Defining data retention policies.

Implementing the ETL process and optimal data pipeline architecture

Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency, and other key business performance metrics.

Create design documents that describe the functionality, capacity, architecture, and process.

Develop, test, and implement data solutions based on finalized design documents.

Work with data and analytics experts to strive for greater functionality in our data systems.

Proactively identify potential production issues and recommend and implement solutions