|
honey the codewitch wrote: Seems like an important feature for a crawler, no?
True but also probably the hardest feature to implement. For example what if the link requires a user login. Or if it requires a selection from the original page. And of course one must eliminate recursive cycles also.
|
|
|
|
|
I usually handle the recursion with a HashTable of crawled URLS.
"Mistakes are prevented by Experience. Experience is gained by making mistakes."
|
|
|
|
|
Yes I know how to handle it. And other problems associated with following links.
|
|
|
|
|
Sure, but get it right if that's what your product does.
And handling recursion is pretty easy on a modern system with a hash table.
In the case of my site, it's nothing fancy. There's a bit of potential recursion is all due to back linking.
There's smoke in my iris
But I painted a sunny day on the insides of my eyelids
So I'm ready now (What you ready for?)
I'm ready for life in this city
And my wings have grown almost enough to lift me
|
|
|
|
|
honey the codewitch wrote: Sure, but get it right if that's what your product does.
I worked on such a product. And getting it right was difficult.
But it did do links.
honey the codewitch wrote: And handling recursion is pretty easy on a modern system with a hash table
Not the point. The point is that following links requires quite a bit of work.
honey the codewitch wrote: In the case of my site, it's nothing fancy.
Yes but you said "get it right" so that means they must handle all of the problems not just yours.
|
|
|
|
|
What are you using to render your files?
On CodeProject, I use Lucene.Net to index and search our content.
I've used Lucene.Net 2, 3, and 4 for various projects and I'm not sure which I like best due to the differences in APIs.
There are versions ported to several languages from the original Java.
Using Stemming Analyzer, you can index by word roots so that 'run', 'runs', 'running', 'ran' all index to the same thing.
I created a custom tokenizer to handle technical documents geared towards programming to weed out some fluff.
It is blinding fast, and looking at the code can be addictive as the algorithms used are quite interesting.
If you are using a 'standard' CMS such as WordPress, Orchard, or most others, they usually have a Search Plugin which is usually based on Lucene.
"Mistakes are prevented by Experience. Experience is gained by making mistakes."
|
|
|
|
|
|
Interesting. We are using Gatsby to render directly from markdown.
There's smoke in my iris
But I painted a sunny day on the insides of my eyelids
So I'm ready now (What you ready for?)
I'm ready for life in this city
And my wings have grown almost enough to lift me
|
|
|
|
|
#Worldle #500 2/6 (100%)
π©π©π©π©β¬βοΈ
π©π©π©π©π©π
https://worldle.teuteuf.fr
"A little time, a little trouble, your better day"
Badfinger
|
|
|
|
|
Wordle 717 4/6
π©β¬β¬π¨β¬
π©π¨β¬β¬β¬
π©β¬π¨β¬π©
π©π©π©π©π©
|
|
|
|
|
Wordle 717 6/6
β¬β¬β¬β¬π¨
π¨π¨β¬π©π¨
π©β¬π©π©π©
π©β¬π©π©π©
π©β¬π©π©π©
π©π©π©π©π©
So many options.
|
|
|
|
|
Wordle 717 3/6
β¬π¨β¬β¬π¨
π¨β¬π©π©π©
π©π©π©π©π©
|
|
|
|
|
Wordle 717 3/6*
β¬β¬π©π¨β¬
π©β¬π©π¨β¬
π©π©π©π©π©
"I have no idea what I did, but I'm taking full credit for it." - ThisOldTony
"Common sense is so rare these days, it should be classified as a super power" - Random T-shirt
AntiTwitter: @DalekDave is now a follower!
|
|
|
|
|
π¨β¬β¬β¬β¬
β¬β¬π©π©π¨
π©β¬π©π©π©
π©β¬π©π©π©
π©π©π©π©π©
Should have had it in 3 ( half asleep ) but I stupidly forgot to include the yellow tile twice !
In a closed society where everybody's guilty, the only crime is getting caught. In a world of thieves, the only final sin is stupidity. - Hunter S Thompson - RIP
|
|
|
|
|
Wordle 717 5/6
β¬β¬β¬β¬β¬
β¬π¨β¬π¨β¬
π¨β¬π¨β¬β¬
β¬π¨π¨π©π¨
π©π©π©π©π©
"A little time, a little trouble, your better day"
Badfinger
|
|
|
|
|
Wordle 717 5/6
β¬π¨β¬β¬π¨
β¬β¬π¨π¨β¬
π¨β¬β¬β¬π¨
π©π¨π©π¨β¬
π©π©π©π©π©
|
|
|
|
|
Wordle 717 5/6*
β¬β¬β¬π¨π©
π©β¬π©β¬π©
π©β¬π©β¬π©
π©π©π©β¬π©
π©π©π©π©π©
Happiness will never come to those who fail to appreciate what they already have. -Anon
|
|
|
|
|
Wordle 717 5/6
β¬β¬π©β¬β¬
π©β¬π©β¬β¬
π©β¬π©π¨β¬
π©π¨π©β¬π¨
π©π©π©π©π©
Ok, I have had my coffee, so you can all come out now!
|
|
|
|
|
Wordle 717 5/6
β¬β¬β¬β¬π©
β¬β¬β¬π¨π©
π©β¬π©π©π©
π©β¬π©π©π©
π©π©π©π©π©
Jeremy Falcon
|
|
|
|
|
Wordle 717 4/6*
π©π¨β¬β¬β¬
π©β¬β¬β¬π©
π©π©π©β¬π©
π©π©π©π©π©
|
|
|
|
|
Wordle 717 4/6
π¨β¬β¬β¬β¬
π©π¨β¬π¨β¬
π©π©π©β¬β¬
π©π©π©π©π©
My mind ignored part of the clue from second word and thus wasted third chance!
|
|
|
|
|
Initially I was thinking why this hasnβt been done yet, I starting typing and then I realized why. I have in mind a general purpose problem solving algorithm that could be applied to RTS games too. To get it working you would need the following.
A list of problems that might get in while you try to achieve something
A list of solutions to each of those problems
It would operate like this:
Start the process PR1 which consists of the things that need to be done to achieve the main goal.
If something unexpected takes place, letβs say obstacle one in the way you have stop PR1, identify what type of problem obstacle one is from the list of problems and pick a solution from the list of solutions to that type of problem. Letβs say the solution is executing process PR2
While executing PR2 a problem may occur again. This means stoping process PR2 and finding a solution to the problem. The solution could be starting process PR3.
If we talk in terms of a RTS game PR1 might consist of units of type 1 UT1, trying to reach destination DT 1, obstacle one could consist of units of type 2 with a different color and with a fire range reaching at UT1, UT2 are located somewhere that UT1 canβt get to.
PR2 could mean airborne units that need to get to UT2. The problem that might get in the way while executing PR2 could consist of ground to air defensive buildings that are found somewhere down the road towards UT2 having the same color as UT2 Iβm not going to cascade any further.
If PR2 is completed PR1 is resumed.
The problem is that in a RTS game things donβt stay as they are too much time. There is a chance UT2 might get moved by the owner from their initial location, that would leave PR2 in mid air.
What are your thoughts?
|
|
|
|
|
This actually reminds me of complex questlines in Bethesda's RPG engines, like Underground Undercover in Fallout 4, as well as the intermingling of the questlines due to competing factions creating twists in the story depending on your actions.
Your method might be an easier way to accomplish it, but it actually might make the problems created by open ended / open world gaming with intermingling quests worse, since it would allow a freer sort of interaction, which leads to smurfing/exploiting opportunities and quest bugs.
Not that your system would have them as a matter of course, but the way it operated it would be difficult to devise a test matrix for it. Let's put it that way.
Edit: I used RPGs as opposed to RTSs as an example because my experience with RTS games is extremely limited.
There's smoke in my iris
But I painted a sunny day on the insides of my eyelids
So I'm ready now (What you ready for?)
I'm ready for life in this city
And my wings have grown almost enough to lift me
|
|
|
|
|
Traditionally RPG worlds are static. If you stop in the middle of a RPG level in a safe area and then leave your computer running for a few hours unattended, when you get back you will find things exactly the way you left them.
If you do the same in a RTS by the time you get back the AI player will have mined everything there is to mine on the map and had destroyed your base ten times over.
|
|
|
|
|
If you leave your character sitting in the open world in the fallout series you will be eaten by a Yao Guai
There's smoke in my iris
But I painted a sunny day on the insides of my eyelids
So I'm ready now (What you ready for?)
I'm ready for life in this city
And my wings have grown almost enough to lift me
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
