We Scraped Ahrefs Evolve 2024 Attendees & Enriched It with Professional Profile Data to Identify Noteworthy People to Network With (Spreadsheet Included) | Enrich Layer
We Scraped Ahrefs Evolve 2024 Attendees & Enriched It with Professional Profile Data to Identify Noteworthy People to Network With (Spreadsheet Included)
My marketing team and I are attending the Ahrefs Evolve 2024 conference in 2 days time. My objective (as the CEO of the company) in attending this conference is really to identify a few brilliant individuals whom I might be able to hire to join Enrich Layer to bolster our marketing team. Just yesterday, I was logging into Micepad, the official platform for the conference, to check out the event agenda and schedule; and I found that the full list of attendees is located within the dashboard for my browsing.
As a product CEO and a software developer, I noticed that the attendees were lazily loaded as you scrolled to the end of the page. Also, I noticed that the full list of attendees can be fetched. The problem is that I lack finer details of attendees. What kind of companies do they work for? How noteworthy are they? Instantly, I realized that Enrich Layer with our Person Lookup Endpoint is the solution I am seeking. I already have the first/last name and the company they are working with; therefore I can pair each attendee with the professional networks profile URL, and then enrich it further with their profile data such as follower count, etc.
I opened up my handy developer tools in Firefox, and scrolled to the end of the page of Micepad Attendee page; checked out the XHR requests that was happening and I saw this request simulated by this curl command:
curl 'https://app.micepad.co/api/web2/getInstanceAttendees' \ --compressed \ -X POST \ -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:132.0) Gecko/20100101 Firefox/132.0' \ -H 'Accept: application/json, text/plain, */*' \ -H 'Accept-Language: en-US,en; \ -H 'Accept-Encoding: gzip, deflate, br, zstd' \ -H 'Referer: https://micepad.co/' \ -H 'Content-Type: application/json' \ -H 'Origin: https://micepad.co' \ -H 'DNT: 1' \ -H 'Connection: keep-alive' \ -H 'Sec-Fetch-Dest: empty' \ -H 'Sec-Fetch-Mode: cors' \ -H 'Sec-Fetch-Site: same-site' \ -H 'Pragma: no-cache' \ -H 'Cache-Control: no-cache' \ -H 'TE: trailers' \ --data-raw '\{ "expressions":[], "moduleid":221220, "searchTerm":"", "offset":4, "apikey":"xxx-censored-xxx", "instanceid":6751, "languageid":7277 \}'
Lo and behold, the attendees were returned in JSON like this:
[ , "bmChoicesIds": [], "realScore": 0, "totalScore": 0, "rank": 13, "allowInitiateText": 1, "allowReceiveText": 1, "allowInitiateVideoCall": 0, "allowAnswerVideoCall": 0, "allowRequestMeeting": 1, "allowAcceptMeeting": 1, "onlineStatus": 0 }, , "bmChoicesIds": [], "realScore": 0, "totalScore": 0, "rank": 18, "allowInitiateText": 1, "allowReceiveText": 1, "allowInitiateVideoCall": 0, "allowAnswerVideoCall": 0, "allowRequestMeeting": 1, "allowAcceptMeeting": 1, "onlineStatus": 0 }, ... ]
With common sense while analyzing the request, you can see that I can basically paginate the entire attendee list by incrementing the offset value from 0 to N.
Here's the parameter set in question:
\{ "expressions":[], "moduleid":221220, "searchTerm":"", "offset":4, // <--- increment this number starting from 0 "apikey":"xxx-censored-xxx", "instanceid":6751, "languageid":7277 \}
So, with some trusty Python for which I barely touch these days, I wrote this up to paginate through Micepad to get the full list of attendees:
if total_records is None:
total_
print(f"Total records: {total_records}")
all_attendees.extend(attendees)
print(f"Fetched attendees so far…")
if len(attendees) == 0 or len(all_attendees) >= total_records:
print(f"Fetched attendees in total")
break
payload['offset'] += 1
print(f"Total attendees fetched: ")
with open('attendees.json', 'w') as f:
json.dump(all_attendees, f,
`
Now I have a list of attendees. But that isn't enough. I want to enrich each of these attendees with their professional networks profile URL and sort these attendees by their follower count.
Enriching attendees with professional networks Profile URL and profile data
It turns out that Enrich Layer has the perfect tool for this. With a person's first and last name, and the company they are working with, there is a very good chance we'll be able to match this person to his/her professional networks Profile URL via the Person Lookup Endpoint. On top of that, it can enrich the matched professional networks Profile URL with the full dataset, all within one API request.
So given that there are 522 attendees and I don't have all day. So I decided to sprint through the attendee list by making API calls to the Person Lookup Endpoint concurrently. I did this with a worker pool of 10 (coincidentally, while doing so, I found a bug within our API endpoint for which we were returning the wrong status code when we failed to match it with any result. This is now being fixed, hah).
Here's how I did it:
`def enrich_attendee(attendee):
api_ ”)}".strip(),
'title': attendee.get('title', ”)
}
max_
while retries < max_retries:
if response.status_ 200:
enriched_
attendee['enriched_data'] = enriched_data
break
elif response.status_ 503:
retries += 1
if retries < max_retries:
print(f"Received 503 error. Retrying ({retries}/{max_retries})…")
time.sleep(2 retries) # Exponential backoff
else:
print(f"Failed to enrich attendee after {max_retries} retries: {attendee['fullName']} - Status code: ")
print(f"Params for failed request: {params}")
else:
print(f"Failed to enrich attendee: {attendee['fullName']} - Status code: ")
break
return attendee
def enrich_attendees():
with open('attendees.json', 'r') as f:
Load existing enriched attendees
try:
with open('attendees_enriched.json', 'r') as f:
enriched_
except Exception:
enriched_
Create a set of already enriched attendee IDs for quick lookup
enriched_ for attendee in enriched_attendees)
Create a lock for thread-safe writing
write_
def enrich_and_save(attendee):
if attendee['userid'] not in enriched_ids:
enriched_
with write_lock:
with open('attendees_enriched.json', 'a') as f:
json.dump(enriched_attendee, f)
f.write('\n')
print(f"Enriched and saved attendee: {enriched_attendee['fullName']}")
return enriched_attendee
else:
print(f"Skipped already enriched attendee: {attendee['fullName']}")
return None
try:
for attendee in attendees:
futures.append(executor.submit(enrich_and_save, attendee))
for future in as_completed(futures):
enriched_
if enriched_attendee:
enriched_attendees.append(enriched_attendee)
except KeyboardInterrupt:
print("\nScript interrupted by user. Aborting remaining tasks and saving progress…")
executor.shutdown( cancel_
finally:
executor.shutdown(
print(f"Enriched attendees. Results saved to attendees_enriched.json")
`
Quite easily done. This was completed in a few minutes.
Exporting to CSV
JSON is not my favourite way to work with files. I much prefer my spreadsheet UI. And my spreadsheet of choice is Google Sheets. So, I decided to export the dataset to CSV for which I can easily open with Google Sheets.
`def export_to_csv():
try:
with open('attendees_refreshed.json', 'r') as f: for line in f]
except FileNotFoundError:
print("Error: attendees_refreshed.json not found. Please run the 'refresh' command first.")
return
csv_
with open(csv_filename, 'w',,) as csvfile: Name', 'Last Name', 'Latest Job Title', 'Current Company',
'professional networks Profile URL', 'Current Company professional networks Profile URL',
'Follower Count', 'Bio', 'Headline', 'Country']
writer.writeheader()
for attendee in attendees:
enriched_ {})
profile_ {}) [])
current_ if experiences else {}
Yeah, it turns out that the Person Lookup Endpoint returns data that has varying levels of freshness. I wanted fresh profile data. So I needed to make another pass on the attendee data via the Person Profile Endpoint with the `use_ parameter to fetch fresh profile data.
Of course, this is completely optional and I really did this because it was cool. This was trivial given that I already have the professional networks Profile URL. Given that this is optional, I will leave this as an exercise to the reader.
Enrichment is really useful
I whipped this up in 2-3H and I barely write code. And within a day, I have a list of 522 attendees to the Ahrefs Evolve 2024 conference I'm attending with detailed profile data.
If I can do this for an attendee list, just imagine what you can do for your business or your CRM.
Anyways have fun f you're attending the event, hit up the marketing team at Enrich Layer via [email protected], and we'll be happy to have a chat with you
