Data Engineer (Streaming & Sensor Data)
Addx Corporation
Pay not stated in the posting
National Security Group
Found 3 weeks ago · Confirmed live 19 hours ago
Skills this job asks for
- AWS
- Python
- RADAR
- Access Control
- Capacity Planning
- Data Architecture
- Infrastructure as Code
- Version Control
Tailor your resume for this job
See which of these skills you have, and tailor your resume for this job.
Sign up freeTailor your resume for this job
Start from your profile and shape your resume to this role.
At a glance
- Type
- Not stated
- Level
- Not stated
- Experience
- Not stated
- Work model
- Not stated
- Schedule
- Not stated
- Education
- Not stated
- Benefits
- Not stated
- Team
- Engineering
Description
Data Engineer: Streaming & Sensor Data
Location: Primarily Colorado Springs, CO; Denver, CO, also considered
We are seeking a Data Engineer to build the data foundation for our machine learning
applications. You will integrate time-series measurements, including radar and OPIR data, from
real-time streams, application databases, and systems with varying extraction capabilities.
You will collaborate with senior technical staff and the ML team to characterize available data,
establish reliable pipelines, and deliver cleaned, standardized datasets. You will help select
technologies and develop an architecture based on measured workloads, application needs,
and operational requirements. Our development environment is primarily Python, with AWS as
the preferred cloud platform.
In this role, you will:
● Understand our data sources. Investigate available interfaces, message formats,
historical coverage, data volumes, arrival rates, and quality limitations. Document
findings to guide implementation and capacity planning.
● Build ingestion pipelines. Develop maintainable workflows for streaming and historical
data across databases, APIs, files, and application-specific interfaces. Implement
configurable filtering to capture the messages and measurements needed downstream.
● Shape the data architecture. Evaluate processing, storage, and access options against
representative workloads, reliability needs, interoperability, operational complexity, and
cost. Document decisions and support the architecture’s evolution as requirements
become clearer.
● Deliver trusted datasets. Establish consistent schemas, timestamps, units, identifiers,
and source metadata while preserving measurement meaning. Implement validation and
traceability, and define dependable access methods with the ML team. Support future
feature, labeling, and training-data workflows.
● Operate reliable systems. Build tested, documented pipelines with repeatable
deployment, monitoring, and appropriate access controls. Handle duplicate, missing,
delayed, and out-of-order data, and implement recovery and reprocessing procedures.
We’re looking for someone with:
● Experience designing, implementing, and operating production streaming and batch data
pipelines and their supporting storage systems.
● Strong software engineering skills, including maintainable code, automated testing,
version control, and deployment.
● Practical understanding of streaming behavior, including filtering, consumer progress,
buffering, failure recovery, and replay.
● Experience organizing time-series, telemetry, sensor, or similarly complex data, with an
understanding of schema evolution, partitioning, and efficient access patterns.
● Demonstrated ability to investigate unfamiliar systems, evaluate technical tradeoffs, and
communicate recommendations while balancing reliability, performance, maintainability,
and cost.
communicate recommendations while balancing reliability, performance, maintainability,
and cost.
Additional experience that would be valuable:
● Python-based data processing and AWS or comparable cloud infrastructure.
● Radar, OPIR, or other sensor measurement data.
● Workflow orchestration, containers, infrastructure as code, and automated delivery.
● Dataset versioning, feature generation, labeling, or training-data preparation.
About Addx Corporation
Addx Corporation