How to find all .tv, .kz domains using C# with Google?

Question

1.00/5 (1 vote)

See more:

Hi.
I need to find all domains tied to country specific domain extension?
Because I need to find out all domains (url and total number) with *.tv, *.kz extensions( tv and kz are just example.) for my statics research paper. Most top domain providers are unwilling to share that info. If make Google search "site:*.tv" then I can list all *.tv domains( of course many duplicates because of subdomains). But still have challenge to scrape domain urls to text list. If it requires programming then I prefer C# which my main programming language
Thank you for your suggestions.

Posted 14-Dec-15 17:31pm

xmans

Updated 14-Dec-15 17:33pm

v2

Add a Solution

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Sergey Alexandrovich Kryukov · Answer 1 · 2015-12-14T18:10:00

Solution 1

The search string would be something like "site:.tv". You need to download HTTP response and scrape obtained HTML, and this needs some HTML parser. It could be, first of all, HTML Agility Pack, or something else. Please see my past answers:
How to get particular data from a url using c#[^],
get specific data from web page[^],
see also How to get the data from another site[^].

—SA

Posted 14-Dec-15 18:10pm

Sergey Alexandrovich Kryukov

Comments

BillWoodruff 15-Dec-15 1:19am

+5

xmans 15-Dec-15 22:05pm

Can how can move to next Google search page ("site:.tv")? Because there are hundreds of pages. I mean how can I loop through all of this pages with minimum effort? Is there anyway to get result of Google search in one simple text list or file?

Sergey Alexandrovich Kryukov 15-Dec-15 22:10pm

Good question. But it's easy to answer.

Well, you can parse the first page and locate... give me a second, I'll try on some query...
You can locate the anchor "Next" or ">" and get it's URL. Use this URL in next HTTP request, and repeat the requests until the page without "Next" (or it's hidden or disabled, check it up). This way, you will visit all the pages.

Didn't I provide the complete answer to your question? Will you now accept it formally?

—SA

xmans 17-Dec-15 3:17am

So far nothing is constructive without any code or link to solution. I already knew what you said I just have trouble to figure out how to make it work.

Sergey Alexandrovich Kryukov 17-Dec-15 11:07am

If you knew it, you would not ask your follow-up questions. Sorry, but who would like to help you after your talking about "nothing is constructive" and the like? Please, you are free to do your own work.
—SA

xmans 30-Dec-15 3:58am

Can you post working code for google search for parsing and moving to next page?

Sergey Alexandrovich Kryukov 30-Dec-15 11:42am

The "Google search" is Google code, not mine. :-)
—SA

How to find all *.tv, *.kz domains using C# with Google?