|
Ajax, the bane of content thieves everywhere.
It is actually easier to get the data from these types of sites you do not have to filter the HTML to get to the data. What you need to do is find the underlying js method that returns the data and call it directly. If the given site allows you to do so. The presence of an RSS feed is a de-facto indication that aggregation is expected. In fact, you could even get the data from the RSS feed as well.
|
|
|
|
|
I handle situations like this by opening the page in an off-screen WebBrowser and reading the control's document's OuterHtml property after the document has finished loading. Ennis' solution will work (and is the preferred one) if an underlying AJAX call exists.
/ravi
|
|
|
|
|
Twitter has a well defined API for retrieving information in a variety of formats. Don't waste time trying to screen scrape when you can get the data easily using the RESTful API.
|
|
|
|
|
I'm building a little puzzle game that can have many (Trillions) of puzzles. Each puzzle can be simplified to a grid that is 6 by 6 where each grid can be one of 8 possible values. Right now I have each puzzle stored as a 36 char string (each char can be one of {"L", "R", "T", "B", "E", "P", "H", "V"} does anyone know a cool trick to take this type of string and compress it down so that it takes less space? Even shaving a few bytes off of it would be a huge help. (It'll make searching for it in the database a lot faster)
|
|
|
|
|
This is NOT string or text compression, it is a very dedicated problem.
8 possible values for a cell means a cell only needs 3 bits, hence a 6*6 board could be stored in 108 bits, i.e. 14 bytes (with 4 unused bits).
|
|
|
|
|
Yeah this is the path I'm taking now. It gets a bit more complicated because while some cells can have all 8 values, some specific cells (Corner cells) can have only 3 of those values, and other rules as well. I guess I'm wondering if anyone has seen a class that can be configured to do this. Or should I build one and submit a CP Article.
|
|
|
|
|
If you give me a grid with in each cell the number of values it can take, I can probably give you a more detailed answer. For now I'd say Range Encoding would take care of oddities such as 3 possible values, but maybe tricks can be used to avoid it.
|
|
|
|
|
if corners can only have 4 or fewer values that saves 1 bit each, hence the whole board now fits in 13 bytes. You'll need strong further restrictions on the possibilities to do much better than that.
The simple rule is:
numberOfBitsRequired = CEIL(base-two logarithm of the number of acceptable board states).
|
|
|
|
|
Luc Pattyn wrote: if corners can only have 4 or fewer values that saves 1 bit each
With 4 corners and each corner allowing only 3 possible values, all 4 corners can be stored using 7 bits (because 3^4 = 81 possible combinations, which is less than the 128 combinations 7 bits can store). So, rather than use 2 bits per corner (for a total of 8 bits), only 7 bits are required. It's only a bit, but that might just help to make it down to the next lower byte if the OP can come up with other compression/encoding techniques to save space.
|
|
|
|
|
there's way too many ifs in your message. In my earlier reply I had 4 spare bits, so shaving one bit of every corner yielded another byte. Why make things more complex than they need to be, the scheme I offered is the most efficient for the situation as it has been described. And I'm with Einstein, keep things as simple as possible, but no simpler than that. And I'll join the scientists who amend the theory as soon as new facts come in that don't fit the current one.
|
|
|
|
|
Nothing wrong with that. In fact, having your reply so clear and having my reply which elaborates on it probably will help the OP to understand both approaches more clearly. Just thought I'd add some additional input to this little brainstorm session.
|
|
|
|
|
I can only hope he is not studying DCT algorithms and the like. 
I don't think he'll bring it well below 13 bytes, unless there is a lot he hasn't told.
And that is the main reason I offered him Shannon, so he can calculate the optimum himself.
|
|
|
|
|
And if all the cells on the edge can only have 5 different values, you can combine each corner cell with an edge cell, giving 4 * 15 possibilities (6 bits), and then you have 5^12 possibilities left for all other edge cells together which fits in 28 bits, giving 85 bits in total.
|
|
|
|
|
harold aptroot wrote: And if all the cells on the edge can only have 5 different values
I don't really follow what you said (e.g., what do you mean by "combine each corner cell with an edge cell"?), but keep this in mind:
some specific cells (Corner cells) can have only 3 of those values
We already know each corner cell can only use 3 (rather than 8) values. I didn't just pick an arbitrary value, if that's what you are implying (I honestly don't know what you mean).
|
|
|
|
|
If the corner cells can have 3 values and the middle cells 8, then I see the pattern "they can hold as many values as there are adjacent cells". The OP did not say it, but he's been a bit slow in providing information anyway. It's just a guess, and that's why there is an if near the beginning of my post.
The combining is .. just combining. Combine a corner cell (3 states) with an edge cell (might have 5 states) to get 15 in total, which is very close to a power of 2.
|
|
|
|
|
Luc and harold made good suggestions... There's one other route that will work if you're generating the boards randomly. A lot of randomly-generated games work by not storing the board/scenario itself, but instead storing the seed for the random number generator.
Usually, you get random numbers by seeding with Environment.TickCount, or something similar, but if you use a specific seed (Which could itself be randomly generated), you're guaranteed to get the same results every time, even with a complex algorithm (As long as you don't multi-thread it).
That way, you're only storing one 4-byte integer.
|
|
|
|
|
An interesting concept, however it mainly makes apparent a 4-byte seed isn't sufficient to generate all conceivable board set-ups: rather than some 2^108 it will produce no more than 2^32 of them, a mere drop in the ocean.
BTW: the OP mentioned trillions, not sure he meant that in the long or short scale of ways (see here[^])
|
|
|
|
|
True, but 2^32 ought to be enough for anybody
|
|
|
|
|
like 640KB? 
|
|
|
|
|
|
It wasn't a very obscure joke, for a site like this
|
|
|
|
|
Haha, yeah, I saw it posted the other day. I wondered if perhaps it was you who posted it.
|
|
|
|
|
Don't think it was me... But hey, it's an old joke
|
|
|
|
|
I'd say nobody will ever need more than 640KB of RAM. 
|
|
|
|
|
for 6*6 cells with no more than 8 states each, not really compression, is it?
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
