Jafsal M A

Senior Python Developer · Technical Lead · Data Engineering & Web Scraping

Kerala, India

Summary

Senior Data Engineer and Technical Lead with 7+ years building large-scale web crawling systems and Python ETL pipelines, delivering production-grade ingestion, transformation, and automated delivery at 100Ks-to-millions-of-records scale. Experienced leading data operations teams end-to-end — from crawling through client-format delivery (SAP XML, E4H XML) — and currently moving into agentic AI, building automation agents on the Hermes framework.

Experience

Python Developer / Team Lead – Data Engineering & Web Scraping

Feb 2024 – Present

PST.AGWeimar, Germany (Remote)

  • Led design and delivery of Python backend tooling for a scraping ecosystem of 100+ production projects, improving ingestion throughput and downstream data readiness for analytics
  • Delivered trade-compliance datasets — customs tariff schedules, tariff measures, and sanctions/denied-party data — sourced across a wide variety of projects and shipped to clients in SAP, E4H, and S4H formats
  • Led the Data Operations team for one year, owning the full ETL project lifecycle — crawling, transformation, validation, and automated delivery — with outputs delivered to clients in SAP XML, E4H XML, and other structured formats
  • Led a team of 4 engineers, conducting code reviews, mentoring, and driving internal tooling and performance optimization POCs
  • Built and scaled ingestion pipelines across distributed storage (ArangoDB, Elasticsearch) to process high-volume multi-source feeds with fault tolerance and schema evolution support
  • Designed and operated ETL workflows using Apache NiFi for ingestion, transformation, routing, and automated validation of heterogeneous data streams; orchestrated ETL workflows with Apache Airflow
  • Implemented automated validation and scenario-based verification (BDD) to detect data anomalies earlier and reduce incorrect downstream records
  • Optimized the automated file-delivery script used to ship customer deliverables on each release, improving delivery reliability and reducing manual intervention
  • Completed onboarding for the company’s move into agentic AI and joined the team building automation agents on the Hermes framework, integrating MCP and Mattermost for operational communication
  • Translated client and stakeholder requirements into scoped technical work, contributing to project planning, architecture documentation, and technical specifications
  • Maintained containerized services on Docker, Scrapyd, and Linux infrastructure, integrating releases into CI/CD pipelines for versioned rollouts and monitoring
  • Participated in Agile ceremonies and delivery tracking via Jira

Software Engineer II – Data Operations and Engineering

Apr 2021 – Jan 2024

Shopalyst TechnologiesTrivandrum, Kerala, India

  • Architected Python and Golang backend systems that processed millions of records daily, improving pipeline throughput and operational reliability
  • Built and operated the core data-collection backbone powering multiple downstream products, including brand landing pages (live, cross-retailer pricing per product) and a brand insights dashboard (competitor benchmarking, market trends, share-of-voice, and custom reports)
  • Ran full catalog, product, and review crawls, plus share-of-voice (SOV) crawls, across web, mobile web, and native app surfaces to keep multi-brand datasets current
  • Reverse-engineered Android applications, including SSL-pinned apps, to discover internal endpoints and integrate RESTful APIs into structured pipelines, using BrowserStack and LambdaTest for cross-device testing and validation
  • Built Appium crawlers that captured product-page screenshots and product data from 5-6 mobile apps on a LambdaTest virtual device across 10-12 locations, for client delivery
  • Designed and enforced automated verification pipelines and data quality checks to ensure correctness across extracted datasets
  • Operated distributed storage and messaging to support high-throughput workflows, using Elasticsearch, Cassandra, and RabbitMQ
  • Used Apache Spark (PySpark) for distributed data processing in the data pipelines
  • Automated operational workflows with Bash, Cron, and Jenkins, reducing manual failures and deployment errors
  • Mentored junior engineers and maintained system stability through proactive monitoring, debugging, and incident resolution

IoT Edge Developer

Dec 2020 – Mar 2021

Tata Consultancy ServicesKochi, Kerala, India

  • Built gateway and automation software for Damen (Dutch shipbuilder) vessel systems, integrating onboard IoT sensor and control data via Azure IoT Edge for reliable real-time processing
  • Architected modular C++ services for Azure IoT Edge to enable reliable, low-latency processing across distributed edge environments
  • Delivered Dockerized services and automated builds using Azure DevOps and Git to support continuous integration and deployment
  • Implemented protocol-heavy solutions with strict validation requirements to ensure compliance and system integrity in regulated environments

System Engineer

Dec 2018 – Nov 2020

Tata Consultancy ServicesChennai, Tamil Nadu, India

  • Owned the full catalog-enrichment pipeline — collection, cleaning, and push — for Grainger.com and Zoro.com (US e-commerce), using Scrapy and BeautifulSoup to pull product attribute data from vendor websites and update the client catalog database
  • Built parsing and feature-extraction logic that improved data completeness and reduced downstream cleaning effort, using regular expressions and pandas
  • Documented crawling workflows, contributed to code review processes, and supported knowledge transfer to maintainable scraping codebases

Projects

Full-Stack Food Ordering Web App

Built a full-stack food ordering web app with AI-assisted tooling, owning UI, backend logic, data persistence, and deployment.

  • Firebase Studio
  • Antigravity

Structural Health Monitoring System

Built a remote system to monitor the structural health of old buildings, ingesting sensor streams into Firebase and delivering real-time alerts via Telegram Bot (undergraduate project).

  • ESP32
  • Raspberry Pi
  • Adafruit VL6180X
  • IP Camera
  • Firebase
  • Telegram Bot

crawlops

Self-hosted, container-native operations tool for Scrapy — deploys crawls as isolated containers, inspects logs and Scrapy stats, and schedules cron or one-time runs (personal side project, MVP).

  • FastAPI
  • Python
  • React
  • TypeScript
  • Vite
  • PostgreSQL
  • Docker Compose

curl-to-scrapy

Built and hosted a free, open-source curl-to-Scrapy converter that turns a curl command into runnable Scrapy code (scrapy shell snippet, spider, curl_cffi check script) entirely in-browser.

  • JavaScript

scrapy-legacy-ssl

Drop-in Scrapy middleware for crawling sites served over outdated TLS or weak cipher suites (personal side project).

  • Python
  • Scrapy middleware

tv-tools

Minimal Android TV launcher and a macOS adb-based TV remote with trackpad support, for controlling a HiSilicon Android TV from a Mac over network adb (personal side project, MVP).

  • Shell scripting
  • SwiftUI (macOS)

Skills

Languages
Python (Advanced), SQL, Golang, Java, C++, Bash
Data Ingestion & ETL
ETL/ELT workflows, Apache NiFi, Apache Airflow, Apache Spark (PySpark), Large-scale ingestion pipelines, Workflow orchestration, Multi-source data acquisition, API reverse-engineering, Android app reverse-engineering, Scrapy spiders, Scrapyd, Web crawling
Anti-bot & Scraping Resilience
Anti-bot countermeasures, Browser fingerprinting, Headless-browser evasion, CAPTCHA bypass, User-agent rotation, Rate-limit handling, Proxy rotation and tunnelling, Dynamic (JavaScript-rendered) content
Web & Parsing
Scrapy, BeautifulSoup, Selenium, Playwright, Appium, curl_cffi, HTML, JavaScript, XPath, CSS selectors, Regular expressions, pandas, RESTful APIs, GraphQL APIs
Traffic Analysis & API Tooling
Fiddler, mitmproxy, Burp Suite, Postman, HTTPie, BrowserStack, LambdaTest
Agentic AI
Hermes Agent framework, MCP (Model Context Protocol), Automation agents
Big Data & Storage
Distributed storage, Partitioning, Indexing, Schema evolution, Elasticsearch, Cassandra, ArangoDB, PostgreSQL, MySQL, MongoDB, RabbitMQ, Time-series databases
Infrastructure & DataOps
Docker, Linux, CI/CD (GitLab, GitHub Actions, Jenkins, Azure DevOps Pipelines), Git, Cron, Jira
Cloud & Platforms
Azure, Azure IoT Hub, Azure IoT Edge, AWS (EC2, IoT Greengrass), Firebase
Data Governance
Data lineage, Data contracts, Scenario-based validation (BDD), QA, PII/compliance, Logging & fault analysis
Methodologies
Agile/Scrum, BDD, System design, Code reviews, Architecture documentation, Training sessions, Project planning & execution
Actively Learning
Databricks (Delta Lake, Unity Catalog), Medallion architecture (bronze/silver/gold), Azure Data Factory, Kubernetes
Spoken Languages
English (Fluent), Malayalam (Native), Tamil (Beginner)

Education

Bachelor of Technology, Computer Science and Engineering

Aug 2014 – Jul 2018

NSS College of Engineering, University of CalicutPalakkad, Kerala, India

Publications

Contact

Email
jfzlma@gmail.com
GitHub
github.com/jfzlma
LinkedIn
linkedin.com/in/jfzlma
Location
Kerala, India