Job Lens

Job postings describe what employers are hiring for, but that information is locked in unstructured text spread across thousands of listings. Job Lens turns a sample of real postings into a structured view of what skills are actually in demand, how that demand shifts by role, and which companies are doing the hiring.

Job Lens is built as a static, offline-first product rather than a live query tool. The skill extraction runs once as a batch pipeline and its output is committed as JSON, and the frontend statically imports that JSON at build time with no runtime API or database. This trades live freshness for a simpler, more reliable architecture. The deployed site has zero runtime dependencies and nothing that can fail at request time.

Job Lens has two independent halves. Offline, a Python pipeline (scripts/extract_skills.py) reads a 1,000-posting sample drawn from a 124k-row LinkedIn postings dataset, calls the OpenAI API once per posting to extract structured skills, and aggregates the results into three output JSON files. Online, a static Next.js App Router frontend imports those JSON files directly at build time and renders them as three data-viz sections, with no server route, database or client-side data fetching involved.

The pipeline sends each posting's title and description to gpt-4o-mini in JSON mode at temperature 0, asking for 5-15 categorized skills per posting (language/tool/framework, soft skill, certification or other) with instructions to normalize variant names (e.g. "Python3" to "Python"). Raw per-posting results are cached to data/output/raw_results.json so the aggregation step can be re-run without re-hitting the API.

During aggregation, skills are grouped for counting by a normalized key (lowercased, punctuation stripped) while keeping the first-seen raw phrasing as the display label, so minor naming variants collapse into one entry without losing readable formatting. A hardcoded list of ~40 known staffing and recruiting agencies is filtered out before counting, since those postings are placed on behalf of an undisclosed end employer and would distort both the skills and companies views. The three output JSONs (top_skills.json, skills_by_role.json, top_companies.json) are committed to the repo and consumed directly by the frontend.

The frontend is a single static page built from three presentational components (TopSkills, SkillsByRole, TopCompanies) on a custom editorial/monospace design system, with a custom OG image and favicon generated in-app.

The main design problem was making per-posting company names usable for aggregation without a canonical company database. The same real employer can appear under several string variants across postings, and recruiting agencies post on behalf of employers they don't name. The pipeline addresses the second problem directly with a maintained agency exclusion list, but does not attempt fuzzy-matching or canonicalizing raw employer name variants, so the top-companies view is only as clean as the source data's naming consistency.

Job Lens processes a third-party job postings dataset, not user-submitted or personal data. Postings are sent to the OpenAI API for skill extraction as part of the offline pipeline; there is no user-facing input and no data collected from site visitors. The OPENAI_API_KEY in the repo's .env is used only for offline pipeline re-runs and is not exposed to the deployed frontend.

Ran the extraction pipeline end-to-end across all 1,000 sampled postings, producing three aggregated views (top skills, skills by role, top hiring companies) with 119 staffing-agency postings filtered out of the company and skill counts. Built and deployed a static Next.js frontend rendering all three views with a custom editorial design system.

No verified visitor or usage metrics are currently available.

Still deciding between leaving Job Lens as a static one-shot analysis or building automation to periodically re-sample postings and re-run the extraction pipeline, which would also require adding tests and a CI pipeline currently absent.