How my webcrawler works

[dreammaker182]

This is how Mark Zuckerberg hacked all of those "facebook" websites with his first project...

[5555555jack]

Google works too! :)

[abhay217]

Search on Toctive.com how to build a web crawler

[5555555jack]

@BakuJupiter mostly because I'm not just giving my application away.

[beautyvakbeurs]

So am I right to say that I can make a crawler that copies pages from an other site to my site instantly?

[5555555jack]

@beautyvakbeurs You could, you would have the static version. However, any buttons or input forms would still be pointing to the old site.

[nandter]

Thanks for posting the video. I have never worked on Web Crawler. Could you point me to some good resources preferably language independent?

[5555555jack]

@nandter Your welcome! I didn't really research it all that much when I made it, I just thought it'd be cool and decided to give it a try. A good place to start is with wikipedia, while I don't trust the information it can give you an idea and there's always a list of references at the bottom that you can browse though.

I don't think youtube will let me put a url in here. So you'll have to google it yourself. Happy crawling!

[nandter]

@5555555jack Thanks! I will start working on it soon!

[5555555jack]

I don't have the code anymore, but I had one expression for getting absolute urls, and I had another for concatenating relative urls so that I could turn them into absolute urls.

[honeyresearch]

Can you post your regex that you are using... why do you use two regex?

Thanks

V$h3r

[sirdanis]

tnx ...wat programming language did you use?

[5555555jack]

This was written in Flex and it's an air application.