Learning web scraping in late 2015 allowed me to quickly create new databases with less effort, I created via scraping car models databases for India, Middle East, Australia, as well as other databases (world countries, mobile phones, etc).
I was thinking whenever I can make the next step and create a car parts database. I started studying several car parts websites and talked with various potential customers, deciding together that www.onlinecarparts.co.uk is one of the best. Website is too big and complex to scrap using online scraping tools, so I had to pay my programmer partner to make a custom scraper, but… I saw this warning:
The data shown here, especially the complete database, may not be copied. It is strictly prohibited to duplicate the data and database and distribute the same, and/or instruct third parties to engage in such activities, without prior consent from TecAlliance. Any use of content in a manner not expressly authorized constitutes copyright infringement and violators will be prosecuted.
Furthermore, a car parts database means overkilling myself. As one-man business, is more profitable to create many small databases (1,000-100,000 rows) and sell at prices from 50 to 500 euro to large number of customers.
There are over 10,000 car make, models and engines, and over 1,000 parts in a car, that makes a total of over 10 million parts, scraping at an average rate of 1 page per second would take several months using my computer only, not enjoyable task, even if completed I would had to sell at a very high price at which number of customers will not justify my effort to update regularly.
So, anyone who is interested in a car parts database, feel free to buy from other data providers such as TecAlliance and don’t keep me busy for few months and also put me in copyright troubles of scraping from websites that prohibit copying data.