How to retreive all tweets in time range in .net

Question

0.00/5 (No votes)

See more:

Hi all,

I need to retreive all the tweets in a time range (say for last seven days), irrespective of the user who made them. Also,I want to store them in database so that I can use them later.How can I do this in .net .

Can anybody please help me. I need this urgently.

Thanks in advance!

Shikha

Posted 4-Aug-10 21:37pm

gupta_shikha

Add a Solution

3 solutions

Solution 2

"I need to retreive all the tweets in a time range (say for last seven days), irrespective of the user who made them."

You surely can't mean all Tweets in a day, there are over 50 million per day - that's an awfull lot of data each day. Perhaps you should think about filtering them in some way.

:)

Posted 4-Aug-10 22:36pm

LittleYellowBird

Comments

gupta_shikha 5-Aug-10 6:04am

Hi Alison,

Yes I meant all tweets, as this is my requirement for now as am going to filter them later according to requirement. I think it should not be impossible. Is it?

Thanks,
Shikha

LittleYellowBird 5-Aug-10 6:54am

Hi, Perhaps not, I'm really not sure. I don't work with Databases, but I just thought that after one week you would have 350 million entries in your Database and that sounds like a lot of data to me, but as I say I don't work with databases so what do I know? Best of Luck with it. :)

Solution 4

The API says this is optional

q: Optional. The text to search for. See the example queries section for examples of the syntax supported in this parameter

e.g

http://search.twitter.com/search.json?&q=twitter[^]

So try just with the since parameter

http://search.twitter.com/search.atom?since=2010-02-28[^]

This will undoubtedly fail though because as Alison has pointed out, it's a very large amount of data to return over the wire. Twitter probably have some resource limit \ timeout options otherwise people running these sort of queries would constantly bring the site down.

You going to have to think about how to buffer the data and retrieve in chunks by using a combination of parameters

Possibly since_id and max_id are best bet, e.g

Chunk 1
http://search.twitter.com/search.atom?since_id=1520000000&max_id=1520100000

Chunk 2
http://search.twitter.com/search.atom?since_id=1520100000&max_id=1520200000

etc etc

Even retrieving small data using lots of requests, you're going to chew up bandwidth & I wouldn't be surprised if you find your IP being blacklisted! Don't know what you could possibly want with 50m inane comments about 'brushing my teeth' and 'about to have breakfast'

Posted 5-Aug-10 3:18am

Dylan Morley

Updated 5-Aug-10 3:23am

v2

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Dylan Morley · Accepted Answer · 2010-08-04T22:35:00

Use the Search API to get the data into XML

http://apiwiki.twitter.com/Twitter-Search-API-Method%3A-search[^]

Read the XML into some objects, e.g

http://blogs.msdn.com/b/roberthorvick/archive/2009/01/27/reading-twitter-data-with-c-and-linq.aspx[^]

Now write them to a database of your choice!