Data Engineer

Job Duties

As part of team, build scalable and fault tolerant distributed Big Data systems to resolve business problems surrounding large scale data warehousing, reporting, and real time analytics.  Develop business logic data pipelines for different teams.  Design and build scalable distributed data solutions using AWS and migrate from existing on-premises distribution to AWS based on the business requirements.  Analyze and manipulate large datasets and find patterns and insights within structured and unstructured data.  Develop Spark applications using Scala/Python, working with data frames and Spark SQL to provide a domain specific language for structure data manipulation.  Write and read complex SQL queries for reporting needs.  Implement data quality checks on all data teams.  Provide technical troubleshooting, data governance, business intelligence, data quality checks, and delivery of optimum business solutions using Big Data technologies.  Optimize performance in the data pipeline to reduce the overall run time.  Explore auto-scaling and implementation to reduce the overall run time in the data pipeline and maximum resource utilization.  Onboard new data sets in the pipeline and identify the data model for easy integration and minimize the impacts.  Monitor data pipelines for production support and fixed data issues on a daily basis.  Plan and collaborate with Hadoop administrators to closely monitor jobs to reduce bottlenecks.  Use of Hadoop, Spark, Hive, Sqoop, Kafka, HBase, Python, Scala, Java, SQL, MySQL, Oracle, Snowflake, and AWS.