Tutorial: How to Build Professional Networks Scraper with Python With a Code Example

professional networks is the most popular social media platform for people to meet one another in a professional setting. I recommend you to read our...

In this tutorial, I will guide you with code to get professional networks profile details from a list of professional networks Profile URL. While this tutorial focuses only on getting the profile details you need, data processing can be done according to your needs.

This tutorial will show you two ways of doing it:

sequential and asynchronous

Setting up prerequisites

• Python 3
• A Enrich Layer API key

For sequential method:

• request

For asynchronous method:

• asyncio
• aiohttp

How to get a Enrich Layer the API credential

You can create a free API Key for Enrich Layer API at Enrich Layer's website. However, with the trial, your credits are limited to 10 credits (1 credit for 1 successful profile request).

If you need more credits, it is only $0.01 per profile ead the introduction to Enrich Layer API here and please send an email to [email protected] for inquiries.

Sequential vs Asynchronous

The asynchronous method gives a shorter overall duration than the sequential method as it sends multiple requests at once while the sequential method only sends one request at a time, each request waiting on the previous one to return a response. However, it also depends on the request execution time that varies according to the latency of the server. As such, this tutorial provides both sequential and asynchronous methods to give comparisons of the code between them.

Let's start with viewing the base code from Enrich Layer's documentation :

` import requests

api_

the platform_profile_

api_ # your api_key

header_ = requests.get(api_endpoint, the platform_profile_url},

print(response.content) # To get all profile details

print(response.json()["first_name"]) # To get profile first name

You can change the "first_name"to otherrespond key` in this Enrich Layer documentation

1. Sequential Method

` import requests , json

from requests.exceptions import HTTPError

api_

api_ # your api_key

header_ Social Network_profile_

'https://www.professionalsocialnetwork.com/in/williamhgates',

'https://www.professionalsocialnetwork.com/in/melindagates',

'https://www.professionalsocialnetwork.com/in/owinfrey'

]

def get_profile_details(likedin_profile_url, session):

try: the platform_profile_url},

response.raise_for_status()

print(f"Response status ({url}): ")

except HTTPError as http_err:

print(f"HTTP error occurred: {http_err}")

except Exception as err:

print(f"An error ocurred: {err}")

response_

return response_json

with requests.Session() as session:

for the platform_profile_url in the platform_profile_list:

try: Social Network_profile_url, session)

print (response["first_name"]) # To get first_name key

print(response) # To get all profile details

print()

except Exception as err:

print(f"Exception occured: {err}")

`

Let's breakdown the code.

As usual, we imported the required library.

Then we use all the variables defined before in the base code except the the platform_profile_url. Instead, we create a new list, the platform_profile_list, to hold the URLs of the profiles we want to scrape. For now, we will use 3 user profiles for demonstration purposes.

Next, we define get_profile_details function to make a GET request to Enrich Layer's the API. We pass in the profile URL as a query param of the request, along with the Authorization header. Then the JSON response is parsed into Python dictionary using json.loads().

Uncomment line with print(f"Response status (\{url\}): ") to print response status (200 for success). Take note that a request with a 200 response code indicates a successful request and 1 credit is consumed.

The code block under with requests.Session() as session will iterate through the the platform_profile_list and print the public_identifier.

Uncomment line with print (response) to print all profile details.

2. Asynchronous Method

` import aiohttp, asyncio, json

from aiohttp import ClientSession

from urllib.error import HTTPError

api_

api_ # your api_key

header_ Social Network_profile_

'https://www.professionalsocialnetwork.com/in/williamhgates',

'https://www.professionalsocialnetwork.com/in/melindagates',

'https://www.professionalsocialnetwork.com/in/owinfrey'

]

async def get_profile_details_async(the platform_profile_url, session):

try: session.request(, the platform_profile_url},

response.raise_for_status()

print(f"Response status ({url}): ")

except HTTPError as http_err:

print(f"HTTP error occurred: {http_err}")

except Exception as err:

print(f"An error ocurred: {err}")

response_ response.content.read()

return json.loads(response_json)

async def run_program(the platform_profile_url, session):

try: get_profile_details_async(the platform_profile_url, session)

print (response["public_identifier"])

print(response)

print()

except Exception as err:

print(f"Exception occurred: {err}")

pass

async def run_async():

async with ClientSession() as session:

await asyncio.gather(*[run_program(the platform_profile_url, session) for the platform_profile_url in the platform_profile_list])

def main():

loop.run_until_complete(run_async())

loop.close()

if name == 'main':

main()

`note:   if you got this error on Mac OS:

urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed

In the terminal, try to run:

pip install --upgrade certifi

If it doesn't work, try to run:

open /Applications/Python\ 3.6/Install\ Certificates.command

Let's breakdown the code.

The async keyword that prepends the function signature tells Python that this function is a coroutine.

The await keyword, such as in session.request(...) and response_ response.content.read(), tell that coroutine to suspend execution and give back control to the event loop, while the operation is awaiting finishes.

A coroutine is similar to generators in Python that consume values instead of producing values. It will pause the execution while waiting for the new data.

In our case, it suspends the execution of get_profile_details_async while the request is being performed: await session.request(...). It is suspended again, while the response is being read by the stream reader: await response.content.read() and json.loads(response_json).

Then, we have the run_program coroutine as a wrapper around the pipeline of getting a response from the API, parsing it to JSON, and printing the results on the screen. It awaits the execution of the get_profile_details_async coroutine.

After that, using the asyncio.gather syntax, we tell the program to schedule all the tasks based on the list of coroutines we provided. This is what allows us to execute tasks concurrently.

Lastly, define main function to run run_async function using asyncio.get_event_loop() and run it under if __name__ == '__main__' block.

3. Challenges

There are still many things you can do beyond this tutorial after retrieving the data you need. You can either store all the data first into any type of file you want and do the data processing, or you can filter the data by adding functions to our code before and only saving the information you need, to make your own personalized the platform automation tool