Job descriptions often contain company information, benefits, and promotional content that aren't relevant to the actual position. This filter uses machine learning to identify and remove unnecessary content while preserving information relevant to the role. It primarily focuses on retaining sentences describing required skills, experience, certifications, qualifications, and other job-specific requirements.
The filtering system uses a fine-tuned DeBERTa-v3-small transformer model trained for binary text classification. The model analyzes individual sections of a job description and classifies them as either relevant or irrelevant to the position. The job title is also provided as context, helping the model distinguish between relevant and irrelevant information based on the specific position and slightly improving classification accuracy.
The model was fine-tuned on a manually labeled dataset of approximately 2,600 sections and evaluated using a group-based train/test split to prevent sections from the same job posting from appearing in both datasets. On the held-out test set, it achieved approximately 96.9% accuracy and a 0.970 F1 score.
Because the system relies on statistical predictions, some sections may be incorrectly classified. The model may occasionally remove relevant information or retain unnecessary content, particularly when a section contains both job requirements and general company information.