Scrape LinkedIn Jobs at Scale

Are you trying to build a Job board? If yes then today I will show you how you can pull job data from websites like Linkedin. For this tutorial, we will use LinkedIn Job Scraping API which will help us scrape job data from LinkedIn without getting blocked.

Requirements

We will use Python 3.x for this tutorial. If it is not installed on your machine then you can download it from here. After this create a folder in which you will keep the Python scripts. I am naming this folder as jobs.

mkdir jobs

Then download the following libraries.

• requests- This will be used for creating an HTTP connection with the LinkedIn Jobs Scraping API.
• Pandas- This will be used to convert JSON data into a CSV file.

Now create a python file inside this folder.

Finally, you have to sign up for the free pack of Scrapingdog. With the free pack, you get 1000 credits which is enough for testing purposes.

Scraping Linkedin Jobs

Before we start scraping the jobs let’s look at the documentation first. We can see that there are four required parameters that need to be passed while making the GET request. For this tutorial we we are going to scrape jobs offered by Google in the USA. To check the geoid of the USA you can head back to the LinkedIn jobs page.

As you can see the geoId of the USA is 103644278.

import requests

# Define the URL and parameters
url = "https://api.scrapingdog.com/linkedinjobs/"
params = {
    "api_key": "Your-API-Key",
    "field": "Google",
    "geoid": "103644278",
    "page": "1"
}
# Send a GET request with the parameters
response = requests.get(url, params=params)
# Check if the request was successful (status code 200)
if response.status_code == 200:
    # Access the response content
    data = response.json()
    print(data)
else:
    print("Request failed with status code:", response.status_code)

This code will return all the jobs offered by Google in the USA. Once you run this code you will find this response on your console.

This response contains job_position, job_link, job_id, company_name, company_profile, job_location and job_posting_date.

It is not necessary to pass the company name all the time. You can even pass the job title as the value of the field parameter.

import requests

# Define the URL and parameters
url = "https://api.scrapingdog.com/linkedinjobs/"
params = {
    "api_key": "Your-API-Key",
    "field": "Product Manager",
    "geoid": "103644278",
    "page": "1"
}
# Send a GET request with the parameters
response = requests.get(url, params=params)
# Check if the request was successful (status code 200)
if response.status_code == 200:
    # Access the response content
    data = response.json()
    print(data)
else:
    print("Request failed with status code:", response.status_code)

For every page number, you will get a maximum of 25 jobs. Now, the question is how to scale this solution to collect all the jobs from the API.

Scaling this solution

By scaling we mean to collect all the jobs offered by Google in the USA. We can easily do it by running an infinite while loop. This while loop will run until the length of the data array comes to zero.

import requests
import pandas as pd

page=0
l=[]
while True:
    page=page+1
    url = "https://api.scrapingdog.com/linkedinjobs/"
    params = {
        "api_key": "Your-API-Key",
        "field": "Google",
        "geoid": "103644278",
        "page": str(page)
    }
    # Send a GET request with the parameters
    response = requests.get(url, params=params)
    # Check if the request was successful (status code 200)
    if response.status_code == 200:
        # Access the response content
        data = response.json()
        for item in data:
            l.append(item)
        if(len(data) == 0):
            break
    else:
        print("Request failed with status code:", response.status_code)
df = pd.DataFrame(l)
df.to_csv('google_jobs.csv', index=False, encoding='utf-8')

This code will keep running until the length of data is zero. break statement will stop the infinite while loop.

• After exiting the loop, it creates a DataFrame df from the list l using Pandas.
• Exports the DataFrame to a CSV file named ‘google_jobs.csv’, without the index, and with UTF-8 encoding. This step will create a CSV file that will have all the jobs offered by Google.