For many years, manual data entry in Excel (sourcing from books, as seen in this video) or manual copy-pasting from websites, was my only way of creating databases. A slow process which limited the size of the databases I could make. Even in this slow process I made about 40 databases in various topics: cars, geography, real estate, gaming, etc, from pure hobby.
Starting from 2015 I offer web scraping services… in ANY field, not limited to above topics. Scraping usually means running a software to visit a list of given pages and copy specific data from each page to put it in a database automatically.
You need just to provide URL of a website that you want to scrap data from, and tell me which pieces of data to extract.
Theoretically I can scrap data from any website, but only websites having the required data in a consistent structure across pages, can produce a good usable database. After automatic scraping, less or more manual work is needed to make database usable.
Simple data scraping service
This apply on websites where each item have own URL and all data is in page HTML code.
There are few tools available online, usually free download but they are limited in functionality, limited in number of pages you can extract and in pages per second, unless you upgrade to paid subscription, which is ridiculously expensive for the number of pages you can extract. Although you can scrap yourself for free (small number of pages), may take few days to learn to use them efficiently. Most people do not have time to learn or money for monthly subscription. I can help you… my programmer partner made in Visual Basic a scraping software comparable with the tools available online, but with no limit in number of pages, this allow me to scrap websites at lower price than you can do yourself.
Price: sum of the following 4 things:
Number of pages to be scraped: up to 1,000 pages = $20, up to 10,000 pages = $50, up to 100,000 pages = $200.
Number of columns (pieces of data to scrap from each page): 50 cents for each column.
Complexity: $0 fee for websites where all items are accessible from an index page, extra $ fee if items are displayed with infinite scrolling, pagination, enter data in search boxes, etc.
Work after scraping: certain websites do not provide data in the format you need, I charge extra fee to manually arrange data in Excel.
Complex data scraping service
This apply on websites having drop-down lists, search boxes, JSON, data is behind login, or other actions are needed to display the data you want to scrap. In this case online scraping tools are useless, so my partner will make a custom data scraper made in PHP or Visual Basic, this may take few days depending by his available time.
Price: usually within $200 to $500 range which I share with my partner, price depends more by complexity of website than by number of pages to be scraped.
For less than 200 records may be faster to copy-paste manually than coding a custom scraping software.
Impossible data scraping
Car classifieds websites usually hide seller phone number and contact email, which can be revealed by clicking a button, this is done specially to prevent scraping and protect emails from being spamming. The only solution is to have a human visiting each page and copy-pasting this hidden data, which require large amount of time. If you are an insurance company willing to do SMS or email marketing and you intend to hire me to make a database of car owners sourcing data from classifieds website, most likely I can’t help you because of time.
Anti-scraping
Some websites look simple to scrap, but they turn complex because of IP blocking, CAPTCHA or other measures made either to prevent someone copying data from them or just to prevent DDOS attacks. If you ask for price before starting the job, you should be prepared for price changes. Need to do part of job to be able to tell final price.
Notes
The main advantage of working with me is that once I create a database I can have multiple people purchasing it, so you will pay just a small part of the cost of scraping (if database is something of my personal interest – cars or real estate of Singapore). If you want to keep private, I can sell it just for you at higher price and not publish on website, but the BIG question is what I should do if a second customer ask me to scrap same website and he agrees to publish on website to get cheaper price? I reserve the right to sell to other people if they ask. If you ask to scrap a website out of my personal interests and unrelated with the fields covered by website, I will not publish it because is unlikely anyone else to purchase it, and you need to pay the full cost of scraping.
The databases published on website include FREE updates for one year, with higher update frequency for products with higher sale volume. But if you ask me to scrap a website privately “just for you” you need to pay for each update, price depending by how much time takes each re-scraping.
Scraping software runs at a speed of 0.5-2 pages per second, depending by website. So I may not able to do very large databases, for example if you want to scrap 1 million records with monthly updates. The limit of how many records I can scrap depends by amount of customers in current month.
Data scraping is legal or not?
Usually scraping is legal, but using scraped data in a public website may be illegal.
Depends… if the data is added by volunteers, or by sellers in classifieds websites, scraping is probably legal. But if authors of website hardworked to compile data from sources like car brochures or manufacturer websites, scraping is probably illegal, especially if you use their data in making your own website or other commercial purpose. Most websites contains dummy data (example: a bunch of cars having +/- 1 horsepower than official value) and if you use data copied from them, they can prove from where you stole data and make a lawsuit against you. BEWARE!
For a moment I became concerned if my European car database sourced from AutoKatalog books is a copyright violation, but I came in conclusion that it is fine, as long as mine is an original compilation with different data structure than the book, and it target online audience, while the AutoKatalog is a book sold in shops targeting car hobbyists. I am doing each year over 100 sales without having a single person worrying about copyright.
In case of America, Year-Make-Model is my original compilation sourced from Wikipedia and 3 more websites, while Year-Make-Model-Trim-Specs is web scraping from Edmunds.com website who is also offering API thus allow other websites using their data, so again is legal.
But, since I created Indian car database in 2015 sourcing data from carwale.com I started being concerned that what I am doing may be illegal.
Country matters: I had many customers in India asking me to scrap data from various websites. However, when someone from Europe or America ask me certain data that I don’t have and I propose him to scrap data from a website, most told me legal issues of web scraping.
Funny case: someone offered me to sell a car database that he claimed to have been creating it by working for 4 months, 8 hours per day, copy-pasting data from a website. From copyright point of view does not matter if you scraped automatically or typed every letter manually, as long you copied from a website your work is not original. He was probably not aware of scraping software. If you wasted few months doing something that could have been done in few hours, you’re an idiot (I was an idiot too doing such jobs before 2015 being not aware of scraping software, but small jobs only) and I am still doing in case of European database because I source data from books (offline sources), making an original product on the web.