Skip to content

web crawler

Featured Replies

Does anybody know how to create one of these for a single specifice website? am looking to get one done but have no idea how to do it!!

k:)

Do you mean a web crawler as in what a search engine uses to index a website?

  • Author

as in a web crawler that searches a web page and tells you whats new thats been added to that page

k:)

If you are using IIS you can use the MS indexing system, and search that using a query.

as in a web crawler that searches a web page and tells you whats new thats been added to that page

There are apps/services out there which will do it...usually the apps cost some money, a lot of the services that run online are free-registration. If you're happy with the latter, you could try WatchThatPage as that's free.

If you wanted an app but didn't want to pay for one, you could probably crib something using Wget and diff...

Do you have permission of the website host to use an automated app on it? Just quite a few webmasters block automated checks on their websites.

Rob.

  • Author

have just registred on wathchthapage.com so will see what happends when its time for the update:)

k:)

  • Author

tried watchthatpage.com but all that seems to do is inform you that there are changes but doesnt show what the changes are!!....unless im doing it wrong!!

k:)

  • Author

anybody else got ant ideas for me?....cant seem to get watchthatpage to work or send me an email!.....just want one that checks for changes once a week and emails me what the new aditions/ changes are:)

k:)

anybody else got ant ideas for me?....cant seem to get watchthatpage to work or send me an email!.....just want one that checks for changes once a week and emails me what the new aditions/ changes are:)

If you have a Unix box (or a Windows which you can put Unix tools on), you could install wget and schedule that to download the web page to a file on a weekly basis. You could then diff the contents of the one you've downloaded, and the previous version, and then have it notify you of the differences if detects (if it detects any).

Wouldn't cost a penny for you to do, but you would have to do some scripting.

Other than that, you'd probably be looking at paying for a solution. :)

Rob.

What exactly do you want it to do kit?

If you fancy drop me a mail when you send me the other info and i'll point you at what I know.

  • Author

hi mark.....yeh i know your going to ask where that other stuff is i said id to ya!!....the joys of running your own bussiness!!.....the web crawler is for this.....http://www.bonavacantia.gov.uk/forms/estate_search.asp?alpha=all

its to go through on a wednesday night just shortly after midnight to tell me what changes or addition have been made when the website is updated.It saves me going thru every page and trying to find what has been added.

K:)

You know me too well :D

As for that, sounds perfectly doable with Wget or a writing something similar if you can't get it for your OS of choice.

  • Author

Wget?? whats that then and what does it do? As you can see im not the best at this stuff!!

k:)

Create an account or sign in to comment

Recently Browsing 0

  • No registered users viewing this page.

Important Information

Welcome to BRISKODA. Please note the following important links Terms of Use. We have a comprehensive Privacy Policy. We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.

Account

Navigation

Configure browser push notifications

Chrome (Android)
  1. Tap the lock icon next to the address bar.
  2. Tap Permissions → Notifications.
  3. Adjust your preference.
Chrome (Desktop)
  1. Click the padlock icon in the address bar.
  2. Select Site settings.
  3. Find Notifications and adjust your preference.