About the ProjectWe are looking for Data Engineer with knowledge of Databricks, Python and Spark who will join a team working on projects from logistics industry. With a strong focus on building Data Lakes and utilising the Databricks platform, the team continuously push innovation to support business growth. By working mainly with Spark and Python, they work closely together to convert raw data into meaningful insights that enable smarter decision-making. . Tech stack: Dataricks, Python, Pyspark, Spark, SQL, AWS, Airflow Work Mode: remote (with occasional travels to Gdansk - once/quarter)Duration of the project: min. 1 yearJob ResponsibilitiesDesign and build reliable data pipelines to efficiently ingest, combine, and transform data from multiple sources, including internal systems, IoT, fleet data, and external providersGuarantee smooth and consistent integration of data into the Data Lake while meeting data quality and integrity standardsApply advanced knowledge of Databricks and Apache Spark to develop, enhance, and optimise data processing workflows within the Data LakeUse Spark to handle large-scale data processing, execute complex transformations, and support data aggregation for analytical purposesTake responsibility for data storage architecture and management in AWS, including organising data across appropriate buckets and zonesPartner with IT administrators to maintain proper access management and ensure strong data security controlsImplement data tokenisation solutions to protect sensitive information and comply with data protection regulationsRegularly monitor Data Lake performance and proactively identify areas for optimisationImprove pipeline efficiency and storage design to increase data access speed and overall system performanceWork closely with the Product Group (data owners) to understand business needs, offer technical expertise, and maintain data quality across the entire data lifecycleCollaborate with data scientists to ensure data accessibility and readiness for analytical and business use casesProvide advanced technical support to Data Lake users and the engineering team, resolving issues related to ingestion, processing, and accessInvestigate, troubleshoot, and fix data-related incidents promptly and effectivelyAdvocate and enforce best practices in data governance, ensuring compliance with data standards and proper documentationKeep thorough and up-to-date documentation of data pipelines, processes, and data lineage for transparency and future referenceExpectationsMinimum 5 years of professional experience as a Data Engineer, with strong emphasis on designing and maintaining Data Lake architecturesExtensive knowledge of AWS services, particularly in data storage and processing using S3, Databricks, and SparkProven hands-on experience with Databricks, Apache Spark, and Python for large-scale data processing and performance optimisationStrong proficiency in big data file formats such as Parquet, Iceberg, and DeltaGood understanding of data protection regulations and best practices for handling personally identifiable information (PII)Familiarity with data visualisation and reporting tools (e.g., Qlik Sense) is an advantageExperience with containerisation technologies and Infrastructure as Code, particularly TerraformPractical knowledge of workflow orchestration tools such as AirflowProficiency in EnglishWe offerOngoing support from a dedicated agent, taking care of your project continuity, client contact, necessary formalities, work comfort and developmentConsultant Development Program – advice on growth planning based on the latest trends and market needs in IT, including consultations with agents and growth mentorsAccess to 7N Learning & Development – a development and educational platform with webinars, a library of articles and industry reports, and regular invitations to one-time and recurring development events – technical, business, and lifestyleSpectacular integration events, both for you (e.g., annual Kick-Off trip, Christmas parties, or Summer Olympics sports events) and for your loved ones (e.g., family picnics, movie premieres)Professional development not only during the project – you can get involved in knowledge transfer to others within the 7N Services offering directed at 7N clientsRelationships and access to the knowledge of the most experienced IT experts in the market – the average professional tenure of our consultants in Poland is over 10 yearsA complete benefits package, including funding for medical care, life insurance, sports cards for you and your loved ones, as well as discounts in stores in Poland and abroadAbout 7NConstantly searching for projects, difficult rate negotiations, lack of development support – sounds familiar? At 7N, you gain not only stability of contracts but also the personal involvement of a dedicated agent who ensures your professional comfort and continuous access to development initiatives. Our mission is to provide stable and rewarding collaborations that drive your success as an IT expert and the success of our clients. We build long-lasting relationships based on Scandinavian values and 30 years of experience creating IT solutions for over 200 organizations.