Lead Data Engineer

about 2 months ago
Full time role
In-person · Durham, NC, US... more

Position Description

FlexGen makes software that makes a difference in the world. With HybridOS our customers can put more battery storage on to the grid quicker and easier and then keep it running effectively. Traditional energy management systems are often custom developed from the ground up each time a customer builds a project. FlexGen applies modern technologies and product development principles to deliver software that gets the job done, is easy to learn and a joy to use. 

 

We are looking for an experienced Data Engineer to architect, implement, and maintain our diverse data infrastructure. The ideal candidate will have a strong background in cloud services and experience working with large-scale data processing systems. You will be responsible for designing and implementing data pipelines, ensuring data quality, and optimizing data processing performance. You will work closely with our data science and web development teams to deliver valuable insights and be a crucial part of our data-driven strategy. 

Major Job Responsibilities:

  • Architect, troubleshoot, and maintain a scalable, reliable, and cost-effective data infrastructure designed to move large timeseries datasets in near-real time for various end uses 
  • Ensure data quality and integrity through continuous data validation, monitoring, and anomaly detection 
  • Optimize data processing performance by leveraging best practices, such as data partitioning, compression, and caching 
  • Develop and maintain data catalog and schema management systems to facilitate data discovery and governance 
  • Collaborate with the team to establish proper software development lifecycle (SDLC) processes, including version control, code reviews, and continuous integration 
  • Mentor and provide guidance to junior team members, fostering a culture of collaboration and continuous learning 
  • The technologies you may use for this role include:
    • On-premises databases: InfluxDB, PostgreSQL 
    • Data formats: Parquet 
    • Message brokers: MQTT 
    • AWS services: Lambda, Glue, Timestream, RDS, Data Lake 
    • Experienced in one or more of the following languages: SQL, Python, Go, Rust 
    • ETL/ELT processes 
    • Data modeling and warehousing concepts 

Position Requirements:

  • Bachelor's or Master's degree in Computer Science, Engineering, or a related field  
  • 5+ years of experience in data engineering, preferably in a cloud-based environment (AWS experience preferred)  
  • Strong understanding of data modeling, data warehousing, and ETL concepts  
  • Experience with streaming and/or timeseries data  
  • Expertise in designing and implementing efficient data models

FlexGen provides equal employment opportunities (EEO) to all employees and applicants for employment without regard to race, color, religion, sex, national origin, age, disability or genetics. In addition to federal law requirements, FlexGen complies with applicable state and local laws governing nondiscrimination in employment in every location in which the company has facilities. This policy applies to all terms and conditions of employment, including recruiting, hiring, placement, promotion, termination, layoff, recall, transfer, leaves of absence, compensation and training. 

FlexGen expressly prohibits any form of workplace harassment based on race, color, religion, gender, sexual orientation, gender identity or expression, national origin, age, genetic information, disability, or veteran status. 

To the extent permitted by law, employees are subject to periodic random drug testing, and post-accident and reasonable suspicion drug and alcohol testing.