In a time where data is worth its weight in gold, Crunchbase is a goldmine. It’s home to thousands of company profiles, their investment data, leadership position, funding information, news and much more. Crunchbase scraping will allow you to get to the gold chunks (the insights you need) and filter out all the debris (all the other information irrelevant to you).
In this article, we’ll walk you through the process of building a Crunchbase scraper from scratch, including all the technical details and code using Python, with a working demo for you to follow along. With that being said, you should also understand that building a Crunchbase scraper is a time consuming task, with many challenges along the way. That is why we will also go through a demo of an alternative approach using Enrich Layer, a paid API-based tool that does the work for you. With both options on the table, you can weigh their advantages and choose the one that best fits your needs.
Here’s a sneak peak at a basic Crunchbase scraper using Python to extract company name and headquarter city from the website.
`import requests from bs4 import BeautifulSoup 'Mozilla/5.0'} 'html.parser')
name_ class_='profile-name') company_ if name_section else 'N/A'
headquarters_ class_='component–field-formatter field_type_text') headquarters_ if headquarters_section else 'N/A'
print(f"Company Name: {company_name}") print(f"Headquarters City: {headquarters_city}") ` Now, to our alternative approach, Enrich Layer. It is a comparably efficient Crunchbase scraping tool and you can pull the same company information using just a few lines of code. The added benefit here is you won’t have to worry about HTML parsing or any scraping roadblocks with Enrich Layer.
`import requests
api_ 'url': 'https://www.professionalsocialnetwork.com/company/apple/', }
print(f"Company Name: {data['company_name']}") print(f"Company Headquarter: {data['hq']['city']}") ` By the end of this article, you'll be familiar with both methods and be able to make an informed decision. So whether you're excited to roll up your sleeves and code your own scraper or you’re after a one stop solution, keep reading to set up your Crunchbase scraper.
Building your Crunchbase Scraper from scratch
Crunchbase contains several data types including acquisitions, people, events, hubs and funding rounds. For this article, we will go through building a simple Crunchbase scraper to parse out a company's description to retrieve as JSON data. Let’s go with Apple for our example.
First, we will need to define a function to extract the company description. The get_company_description() function searches for the span HTML element that contains the company’s description. It then extracts the text and returns it:
def get_company_description(raw_html): description_ \{"class": "description"\}) return description_section.get_text( if description_section else "Description not found"
This sends an HTTP GET request to the URL of the company profile you want to scrape, in this case, Apple’s profile. Here’s what the full code looks like:
`import requests from bs4 import BeautifulSoup
def get_company_description(raw_html):
Locate the description section in the HTML
description_ {"class": "description"})
Return the text if found, else return a default message
return description_section.get_text( if description_section else "Description not found"
URL of the Crunchbase profile to scrape
Set the User-Agent header to simulate a browser request "Mozilla/5.0"}
Send a GET request to the specified URL
Check if the request was successful (status code 200)
if response.status_ 200:
Parse the HTML content of the response using BeautifulSoup "html.parser")
Call the function to get the company description
company_
Print the retrieved company description
print(f"Company Description: {company_description}") else:
Print an error message if the request failed
print(f"Failed to retrieve data. Status Code: ") ` This script does the trick for pulling Apple’s company description from Crunchbase. Depending on your experience and what you are looking for, things can get a lot trickier. Handling large volumes of data, managing pagination, bypassing authwall mechanisms, there are a lot of hurdles along the way. Keep in mind that you will have to:
-
Perform this action for every single field you’re interested in.
-
Stay updated with any modifications in the web page. Even a small change in how a field is presented in the website can result in a minor or a significant tweak in the scraping logic.
Note: Check the website’s terms of service and robots.txt file to ensure you're scraping responsibly and within legal limits.*
Why is building a Crunchbase Scraper challenging?
Building your own Crunchbase scraper is a viable option, but before you go Gung-ho, be aware of what challenges await you.
Accuracy and completeness
Your efforts will be meaningless if the extracted data is false. Manually scraping raises the margin of error, and the code may overlook important data if the page doesn't fully load or if some content is embedded in iframes or external resources.
Crunchbase's structure and changes
Parsing the HTML of a webpage to extract specific data fields is a basic step in scraping. Crunchbase's HTML is complex, with dynamic elements and multiple layers of containers. It is a task in itself to identify and target the right data. This added with the website’s changing structure can make your job tenfold tougher.
Handling authwalls and anti-scraping mechanisms
Crunchbase protects most of their data behind an authwall and will require login credentials or a premium account. Handling login sessions, tokens, or cookies in the scraper manually makes the task more complex, especially for maintaining these sessions across multiple requests. Similarly, Crunchbase uses bot detection systems and rate-limits requests. You run a risk of getting blocked, and bypassing these protections means implementing techniques such as rotating proxies or handling CAPTCHAs, which is easier said than done.
Building your own Crunchbase scraper gives you flexibility and a sense of accomplishment, but weigh that against the challenges involved. It demands deep technical expertise, constant monitoring and effort to get the data you want. This is without mentioning how time-consuming and prone to errors the process can be. Consider whether the effort and maintenance are truly worth it for your needs.
The hassle-free way to set up a Crunchbase Scraper
Phew uilding your Crunchbase Scraper from scratch sure is some serious work. Not only do you have to dedicate a lot of your time and effort, but also keep an eye out for potential challenges. Thank god Enrich Layer exists ake advantage of Enrich Layer’s endpoints and get all the data you could ever wish for in JSON format. And since Crunchbase only provides public data available on the company, there is no data out of your reach. Any private information scraping attempt will result in 404. Rest assured, you will never be charged for a request that returns an error code.
Enrich Layer provides you with a list of standard fields under the Company Profile Endpoint. You can see a full example of any response in the documentation on the right-hand side below the request that generated it. Enrich Layer has the ability to scrape the following fields at your request:
-
categories -
funding_data -
exit_data -
acquisitions -
extra
Each of these fields that you request comes at an additional credit cost, so choose only the parameters you require. But when you do need them, Enrich Layer puts them a single parameter away ow that we are familiar with Enrich Layer, let's walk through a working demo. We'll include two examples, for Postman and then for Python.
Crunchbase scraping with Enrich Layer via Postman
Step 1: Set up your account and get your API key
Create an account with Enrich Layer and you’ll be assigned with a unique API key. Enrich Layer is a paid API, and you would need to authenticate every request with a bearer token (your API key). You will also receive 100 credits if you signed up with your work email, 10 credits if you used personal email. Then you can start experimenting immediately ere’s what your dashboard should look like.
From here, you can scroll down and choose to work with Person Profile Endpoint or the Company Profile Endpoint. The Person Profile Endpoint is a useful tool if you're looking to scrape the platform. Check out How to Build a profile data Scraper for further details. For this use case, we’ll just be working with the Company Profile Endpoint.
Step 2: Run Postman and set your bearer token
Go to [Enrich Layer's collection in Postman](https://pxlcl.co/Enrich Layer-postman-collection), click on the Company Profile Endpoint doc and find the orange button that says "Run in Postman" and click it. Then click "Fork Collection" and log in however you like. It should look something like this. We have a full tutorial on how to set up Enrich Layer API in Postman.
 and limit it to Enrich Layer. You can do this from the Variables tab or from the pop-up that appears when you start typing into the "Token" field. Name the token to your liking, or just go with the name, Bearer Token.
Verify that the Authorization type is set to "Bearer Token" and that you have typed } into the Token field and click Save in the upper right-hand corner. Remember to click Save!! Your page should look like this:
![Add Bearer Token as the Authorization type](https://lh7-rt.googleusercontent.com/docsz/AD_4nXfvC_3ei0sPFVYpl84jIu-8jDekEP1YH8zHnc97OeXpE3aaoSpCagcTLrtxSrQwA8HbAfq27R05VeZ2oyEDjW092RHSonrlaZawl_8e6pRrWDoEMFGESHFC5BxCD_TkJ4CHRcrzXubqi81SjzCnDqUK210?
Step 3: Navigate to your workspace
On the left-hand side, under "My workspace", go to your collection and then the Company API. You will find a list of options on the dropdown menu, but here’s what you will need to know:
-
Company Profile Endpoint: Enriches company profile with Crunchbase data like funding, acquisitions, etc. You will need to use the company’s professional networks profile URL as input parameter to the API.
-
Company Lookup Endpoint: Input a company’s website and get its the platform URL.
-
Company Search Endpoint: Input various search parameters and find a list of companies that matches that search criteria, and then pull Crunchbase data for these companies.
![The various company-related endpoints](https://lh7-rt.googleusercontent.com/docsz/AD_4nXdLHSPUp1Rox76urpfU55qri2yo1EGKUfHjYnTwTyWO9NSSbOJ7KvcwruhKDcSrQZQ_25H833tfSbyx1BA77fkx5JapRgvFUkQJxXv5GAh2JltEnnMcc0XaGR_TGU-ou7FFQIEnxPFV2kkt4QWVFnHBg3a2?
Step 4: Edit your params and send o to Company Profile Endpoint and from there, you can uncheck some of the fields if you want or modify others. For instance, you might want to change use_cache from if-present to if-recent to get the most up-to-date info, but maybe you don't need the acquisitions information this time.
![Choose the relevant fields that you need. Some cost extra credits.](https://lh7-rt.googleusercontent.com/docsz/AD_4nXdCjSyWAzdWkb-N-NL6iMkgAo-io8kGVkj2NGJeZhiSZWBjF5WfZEk_ICMNc_I9hlUTsG9F3MnjrUcFZV7aUVnfUvbaorNQ7a00Xy5HWPegSnvuFb39zxePQBvIQo4JeAO3Uq3hmmuBVL56e0X-6BEZAZ-4? you've modified all the fields to your liking, click the blue "Send" button in the upper left-hand corner. Your output should look something like this.
) ` Now, what you get is a structured JSON response that includes all the fields that you have specified. Something like this:
"the platform_internal_id": "162479", "description": "We're a diverse collective of thinkers and doers, continually reimagining what's possible to help us all do what we love in new ways. And the same innovation that goes into our products also applies to our practices -- strengthening our commitment to leave the world better than we found it. This is where your work can make a difference in people's lives. Including your own.\n\nApple is an equal opportunity employer that is committed to inclusion and diversity. Visit apple.com/careers to learn more.", "website": "http://www.apple.com/careers", "industry": "Computers and Electronics Manufacturing", "company_size": [ 10001, null ], "company_size_on_the platform": 166869, "hq": \{ "country": "US", "city": "Cupertino", "postal_code": "95014", "line_1": "1 Apple Park Way", "is_hq": true, "state": "California" \}, "company_type": "PUBLIC_COMPANY", "founded_year": 1976, "specialities": [ "Innovative Product Development", "World-Class Operations", "Retail", "Telephone Support" ], "locations": [ \{ "country": "US", "city": "Cupertino", "postal_code": "95014", "line_1": "1 Apple Park Way", "is_hq": true, "state": "California" \} ] ...... //Remaining Data }
Great ongratulations on your journey from zero to data
Is any of this legal?
Yes, scraping Crunchbase is legal. The legality of scraping is based on different factors like the type of data, the website’s terms of service, data protection laws like GDPR, and much more. The idea is to scrape for publicly available data within these boundaries. Since Crunchbase only houses public data, it is absolutely legal to scrape by operating within the Crunchbase Terms of Service.
Final thoughts
A DIY Crunchbase scraper can be an exciting project and gives you full control over the data extraction process. But be mindful of the challenges that come with it. Facing a roadblock in each step can make scraping a time-consuming and often fragile process that requires technical expertise and constant maintenance.
Enrich Layer provides a simpler and more reliable alternative. Follow along with the steps and you can access structured company data through an API without worrying about any roadblocks. Dedicate your time by focusing on using the data and leave the hard work and worry to Enrich Layer e'd love to hear from you f you build something cool with our API, let us know at [email protected] nd if you found this guide useful, there's more where it came from - sign up for our newsletter