Jafsal M A.

Senior Python Developer & Technical Lead – 7+ years

7y 8m 18d exp

Kerala, India · PST.AG (Remote)

I build web crawling and data pipelines at scale, turning messy, high-volume, multi-source data into reliable, structured feeds for downstream analytics.

Over 7+ years I've specialised in:

→Large-scale Scrapy/Scrapyd fleets, anti-bot evasion, and API reverse-engineering
→Distributed ingestion pipelines (ArangoDB, Elasticsearch, Cassandra) with fault tolerance and schema evolution
→ETL/ELT orchestration (Apache NiFi), with PySpark and Airflow in active use
→Leading and mentoring engineering teams (currently leading 4) through code review, architecture, and delivery

Most recently at PST.AG (Germany, remote), I've also been building agentic AI tooling — automation agents in the Hermes framework with MCP integration — alongside the core scraping/data engineering work.

I'm always happy to connect with people working on data extraction, anti-bot systems, or distributed data infrastructure.

Get in Touch Download CV (PDF)

Professional Experience

7 years, 8 months, 18 days

Career tenure since 18 December 2018 across data operations leadership, distributed ingestion systems, and high-throughput pipelines.

Python Developer / Team Lead – Data Engineering & Web Scraping

Current Role

PST.AG

Feb 2024 – Present

Weimar, Germany (Remote)

Led design and delivery of Python backend tooling powering 400K+ daily executions, improving ingestion throughput and downstream data readiness for analytics
Led the Data Operations team for one year, owning the full ETL project lifecycle — crawling, transformation, and automated delivery — with outputs delivered to clients in SAP XML, E4H XML, and other structured formats
Built and scaled ingestion pipelines across distributed storage (ArangoDB, Elasticsearch) to process high-volume multi-source feeds with fault tolerance and schema evolution support
Optimized the automated file-delivery script used to ship customer deliverables on each release, improving delivery reliability and reducing manual intervention
Implemented automated validation and scenario-based verification, increasing detected data anomalies and reducing incorrect downstream records by measurable rates
Designed and operated ETL workflows using Apache NiFi for ingestion, transformation, routing, and automated validation of heterogeneous data streams
Contributed to project planning and execution, translating client and stakeholder requirements into scoped, deliverable technical work
Completed onboarding for the company's transition into agentic AI and joined the team building automation agents on the Hermes framework, integrating MCP and Mattermost for operational communication
Led a team of 4 engineers, conducting code reviews, mentoring, and spearheading internal tooling and performance optimization POCs to raise team productivity
Maintained containerized services on Docker and Linux infrastructure, integrating releases into CI/CD pipelines for versioned rollouts and monitoring
Participated in Agile ceremonies and delivery tracking via Jira to maintain sprint velocity and transparency

PythonApache NiFiArangoDBElasticsearchSAP XMLE4H XMLHermes FrameworkMCPMattermostDockerLinuxGitLabJira

Software Engineer II – Data Operations and Engineering

Shopalyst

Apr 2021 – Jan 2024

Trivandrum, Kerala, India

Architected Python and Golang backend systems that processed millions of records daily, improving pipeline throughput and operational reliability
Reverse-engineered Android applications to discover internal endpoints and integrate RESTful APIs into structured pipelines, using BrowserStack and LambdaTest for cross-device testing and validation
Designed and enforced automated verification pipelines and data quality checks to ensure correctness across extracted datasets
Operated distributed storage and messaging to support high-throughput workflows, using Elasticsearch, Cassandra, and RabbitMQ
Automated operational workflows with Bash, Cron, and Jenkins, reducing manual failures and deployment errors
Mentored junior engineers and maintained system stability through proactive monitoring, debugging, and incident resolution

PythonGolangElasticsearchCassandraRabbitMQReverse EngineeringBrowserStackLambdaTestBashCronJenkins

IoT Edge Developer

Tata Consultancy Services

Dec 2020 – Mar 2021

Kochi, Kerala, India

Architected modular C++ services for Azure IoT Edge to enable reliable, low-latency processing across distributed edge environments
Delivered Dockerized services and automated builds using Azure DevOps and Git to support continuous integration and deployment
Implemented protocol-heavy solutions with strict validation requirements to ensure compliance and system integrity in regulated environments

C++Azure IoT EdgeDockerAzure DevOpsGit

System Engineer

Tata Consultancy Services

Dec 2018 – Nov 2020

Chennai, Tamil Nadu, India

Developed Python ETL crawlers using Scrapy and BeautifulSoup to extract, clean, and transform e-commerce data supporting downstream analytics and ML feature pipelines
Built parsing and feature-extraction logic that improved data completeness and reduced downstream cleaning effort, leveraging regular expressions and pandas for transformation
Documented crawling workflows, contributed to code review processes, and supported knowledge transfer to maintainable scraping codebases

PythonScrapyBeautifulSouppandasRegular ExpressionsETL

Technical Skills

Comprehensive technical capabilities, production data engineering toolsets, and active frameworks.

Data Ingestion & ETL

ETL/ELT workflowsLarge-scale ingestion pipelinesWorkflow orchestrationMulti-source data acquisitionAPI reverse-engineeringScrapy spidersWeb crawlingAnti-bot countermeasures

Agentic AI

Focus

Hermes Agent frameworkMCP (Model Context Protocol)Automation agents

Actively Building

Scaling

PySparkApache AirflowDatabricks (Delta Lake, Unity Catalog)Medallion architecture (bronze/silver/gold)Kubernetes

Languages

Python (Advanced)SQLJavaGolangC++Bash

Infrastructure & DataOps

DockerLinuxCI/CD (GitLab, Jenkins, Azure DevOps)Git

Big Data & Processing

Distributed storagePartitioningIndexingSchema evolutionNoSQL systems (Elasticsearch, Cassandra, ArangoDB)

Cloud & Platforms

AzureAzure IoT HubAzure IoT EdgeAzure Data Factory (learning)Databricks (learning)Firebase

Data Governance

Data lineageData contractsScenario validationQAPII/complianceLogging & fault analysisData validation

Methodologies

Agile/ScrumSystem designCode reviewsArchitecture documentationTraining sessionsProject planning & execution

Web & Parsing

ScrapyBeautifulSoupHTMLJavaScriptRegular expressionspandasRESTful APIsGraphQL APIsTime series databases

Projects

Systems and applications highlighting IoT data ingestion, rapid prototyping workflows, and full-stack solutions.

Structural Health Monitoring System

Built a remote structural health monitoring system ingesting sensor streams and delivering real-time alerts via Telegram Bot.

Research Paper

PDF

"Different Approaches To Crack Monitoring of Buildings Using IoT" — IJRASET Publications (Aug 2018)

ESP32Raspberry PiAdafruit VL6180XFirebaseTelegram Bot

Rapid Prototyping & AI-Assisted Tooling (Vibe Coding)

Developed a full-stack food ordering web app using rapid prototyping workflows, owning UI, backend logic, data persistence, and deployment.

Firebase StudioAntigravity

Web Applications Portfolio

Delivered multiple responsive web applications showcasing fast prototyping, UI/UX implementation, and adaptation to AI-assisted development workflows.

Web developmentUI/UXResponsive design