Job Market Crawler & Analytics

I've always been fascinated by the gap between what job boards tell you and what the market actually pays. Every week, thousands of developers scroll through ITviec listings asking the same question: "Which skill should I learn next?" The usual advice is a mess — bootcamps push whatever is trendy, forums repeat stale opinions, and actual salary data stays locked inside private offers.

So I built a system that connects the dots. The Job Market Crawler doesn't just scrape job postings — it cross-references local Vietnamese demand against global compensation benchmarks to find the real opportunity gaps.

The Triangular Engine

The core of the system is a triangular correlation engine that tracks three dimensions simultaneously:

1. Local demand — scraped from live ITviec job postings across Backend, Fullstack, and Java roles.
2. Global compensation — parsed from Stack Overflow's 2025 Developer Survey covering 90,000+ respondents.
3. Industry trajectory — Kaggle datasets mapping AI adoption curves through 2030.

Every skill gets an Opportunity Gap score: Gap = Global_Median_Salary − Local_Estimate. Skills with a large positive gap and strong local hiring signal are flagged as high-ROI targets — the moves that actually advance a career, not just follow a trend.

Surviving the Crawl

Scraping Vietnamese job boards is harder than it sounds. ITviec runs behind Cloudflare, which meant I couldn't just send a simple GET request. I used curl-cffi with session cookie management to mimic real browser traffic and avoid 403s. Even then, I had to handle pagination carefully and deduplicate listings across overlapping search categories.

The real challenge was parsing salary data. Vietnamese job descriptions don't follow a standard format — some list monthly VND, others quote annual USD, and many bury the range inside dense paragraphs of Vietnamese text. I built a regex-based extraction engine that handles both currencies, detects ranges vs. fixed offers, and normalizes everything to monthly USD for comparison.

What the Data Revealed

From 131 job postings analyzed, the numbers told a clear story:

| Category | Job Mentions | Market Presence | |---|---|---| | Data & Messaging | 85 | 64.9% | | Infrastructure & DevOps | 77 | 58.8% | | Architecture | 55 | 42.0% | | Spring Ecosystem | 29 | 22.1% | | Java Core | 10 | 7.6% |

Vietnamese employers care far more about infrastructure and data engineering skills than raw language proficiency. The real bottleneck isn't knowing Java — it's having production experience with distributed systems. The Opportunity Gap analysis confirmed that developers who invest in DevOps, data pipelines, and system design gain salary leverage that pure language mastery never provides.

The Intelligence Pipeline

The system is built as a modular pipeline:

• Crawlers — Pluggable scraper modules per job board, designed for easy expansion beyond ITviec.
• Analytics — Skill extraction, trend analysis, and Kaggle global intelligence correlation.
• API — FastAPI backend serving intelligence reports to a Next.js dashboard.
• Visualization — Matplotlib heatmaps and Seaborn charts that visualize market demand hierarchies.

What I Learned

This project reinforced something I've long suspected: the most valuable skills aren't the ones with the most job postings — they're the ones where global demand exceeds local supply. Standard job boards optimize for filling roles today. They don't tell you which skills will pay a premium tomorrow.

By correlating global salary data with local hiring patterns, the system detects these disconnects automatically. It produces ranked Opportunity Gap reports and visual heatmaps that show exactly where the leverage is. For a developer in Vietnam deciding between learning another framework or mastering infrastructure, that signal is worth more than a hundred recruiter emails.

What's Next

I want to expand the crawler to cover more Vietnamese job boards and add real-time tracking of how skill demand shifts month over month. The long-term vision is a market intelligence feed that updates automatically as new listings appear — a live radar for career arbitrage.