professional networks sued Mantheos, a real-time professional networks Scraping API service, on 1st February 2022 for fraudulently scraping millions of profiles. In a way, Mantheos is a competitor of ours, and I have always been curious about how they can scrape private professional networks profiles at scale. Finally, I understood how they do it thanks to the legal docket of professional networks's lawsuit against Mantheos and their founders.
I am sure I am not the only one curious, so I decided to write this article to tell you how to crawl millions of private user profiles the way Mantheos did (or is still doing.).
To scrape million of private user profiles like Mantheos, you will need:
- Residential IP proxies. There are plenty of services for which you can procure such proxies.
- Empty debit cards with privacy features or fake names (According to professional networking platforms, Mantheos founders used many debit cards with fake names)
Once you have the above:
- Shuffle through the residential IP addresses and create fake the platform accounts. One account per residential IP. Remember to stick to the same corresponding IP for each new account created for the following steps.
- For each new the platform account, sign up for a trial for a the platform Sales Navigator trial with the blank debit card.
- Use the platform's internal API to fetch private professional networks profile data. You can use a library [like this](https://github.com/tomquirk/the platform-api) which has reverse-engineered the platform's internal API.
- Rotate between the new accounts and scrape as many profiles as you can. When accounts get banned, rinse and repeat steps 1-4.
- Profit (?)
Why use a private debit card (with fake names?)
the platform accounts upgraded with a premium sales navigator come with a higher threshold of API rate limits before the account gets blocked/banned.
Why do you need residential IP proxies?
the platform uses IP addresses as a fingerprint to identify an account to be a unique person.
What is next?
Wait for the platform to sue you. But in all seriousness, don't do this. Scraping private user profiles with such a fraudulent method violates all kinds of laws and agreements. The data would not be GDPR or CCPA compliant.
Use Enrich Layer instead. We scrape public user profiles only, complying strictly with PDPA and CCPA regulations. Besides, you get so much more than just professional networks profile data. With Enrich Layer, you get funding data of companies, contact info of individuals, etc.
Switch to Enrich Layer today, and don't get yourself sued by the platform. Instead, send us an email at [[email protected]](mailto: [email protected]), and we will be happy to match your package with Mantheos.