Handling Professional Networks Public Profiles that exist but cannot be scraped

So you are trying to scrape professional networks profiles in bulk. The chances are that you are crawling the public profiles with a proxy (in a very...

So you are trying to scrape professional networks profiles in bulk. The chances are that you are crawling the public profiles with a proxy (in a very inefficient manner, as discussed in my other post on building your own professional networks Profile Scraper). If so, then you will come across a weird situation for which a public profile for a user cannot be found, but the profile exists if you log into professional networks.

This is because such user profiles do not have a public profile because their profile visibility is set to Private.

You can verify that this is profile is indeed not scrapeable by searching for that profile on Google. If even Google does not have you in their search index, that means that profile does not have a public profile page, and hence it is un-scrapeable.

How does our the platform Person Profile endpoint deal with such profiles?

This week, we made an update to improve our the platform Person Profile API endpoint with a precise 404 error status code when a profile lacks a public profile. It was previously returning a general 500 error status code.

You can read up more about status code handling with Enrich Layer here.