|
I want to build a web crawler that will take a list of urls and search those urls for events happening. I want the crawler to pick up details such as address, image urls, description of event, title of event. And anything else that would be useful for sombody wanting to know about an event. I would like to write this program in java or nodejs. Doing this project quickly and simply is important.
I have checked out nutch the java framework, but I had a difficult time getting going with it quickly. I want my web crawler to be up and running by the end of the week, so the simplest quickest solution is important.
What frameworks should I use and/or what advice do you have to complete such a project?
|
|
|
|
|
Member 13566357 wrote: what advice do you have to complete such a project?
Be prepared for years of work; what you are asking for is far beyond a few simple classes. You would need to read each url, break it down into all its different parts and somehow analyse the content to identify each event (whatever you mean by that). You would then need to follow links from the event to extract any other relevant details. Just take a look at a few websites and see how they advertise events, each one is different.
|
|
|
|
|
Thanks for the input. But there is a huge difference between a few simple classes and 'years of work'. I think that maybe my problem was not defined well. But mainly I was hoping to get advice for a simple, quick to setup webcrawler with nodejs or java. Like a framework or a tool or somthing. As mentinoed I checked out nutch, but it seems overkill. I am not trying to scrape the whole web and I dont want to have to type like 300 characters into a terminal to start it up. I want to define in the beginning maybe 5 urls to scrape in the beginning and slowly but surely add to that url list. Any helpfuls suggestions are appreciated!
|
|
|
|
|
One URL or a million, the issue is exactly the same. Do you understand how to extract the information you want from any web page?
|
|
|
|
|
Ive never built a web crawler so in the context of a webcrawler no. In the context of the DOM, then I was planning to just cycle through those elements and take the data. But yeah I havent gotten that far yet, I thought this would be a good first step. I was also looking at elastic search, and wondering if somehow that might be useful.
|
|
|
|
|
Well that is easy enough to test. Write a small piece of code to pull a page from any website and go through each element in the DOM. Now you have those elements how do you identify these events that you are looking for?
|
|
|
|
|
I havent gotten that far yet. I dont really know. There are a lot of tools out there that help developers with these kinds of problems. I dont think I am the first person who wants to crawl websites looking for events, so it seems like there might be some already made tools either a java dependency I can put into my pom file or some npm package that will help me to solve that issue. If there is no such dependency, then I guess I will have to make it. If that is the case, the spontaneously, dont know if this is a good idea, because I came up with it 5 seconds ago. But You could create a bunch of lists of keywords that can be used as titles for the different data points I am interested with for example. Date could have a title on a website "date","Day of event", "Time", and so fourth. And then when you parse you looked for those key words, and then if you find it then you take the text that is nearby. Then using regular expressions you identify and remove any html tags. you could also have another list of regular expressions that identify if the text following the key word matches the regular expression, so in the case of date, you have like 50 regexes all identifying a date, so that you make sure to get a date. But yeah came up with that in 5 seconds. All I really want is events for a particular city on a particular date, maybe there is an API that takes care of that? Google doesnt, last time I checked.
|
|
|
|
|
Like I said try some basic tests; go and look at some websites and see how they identify the sort of events you are looking for. The issue is really not about getting the web pages, that part is relatively simple. The issue is how you analyse the data and identify the parts you are interested in, and that is the complicated part.
|
|
|
|
|
Yeah agreed. But it seems like maybe there are some dependencies, api, or somthing else that could help. It is that kind of feedback I am hoping to get.
|
|
|
|
|
provide Huffman coding for image compress using java
|
|
|
|
|
That is a complete search query for Google to get the resource you are looking for, or you can always search for a complete article in the top right corner, and find the resources.
GitHub - mpomery/huffman: Huffman Compression Implimented in Java
The sh*t I complain about
It's like there ain't a cloud in the sky and it's raining out - Eminem
~! Firewall !~
|
|
|
|
|
This site is not a free search service.You have to learn how to use Google.
Quote: provide Huffman coding for image compress using java
Your question is not even workable. Image compression is much more than just Huffman coding.
Patrice
“Everything should be made as simple as possible, but no simpler.” Albert Einstein
|
|
|
|
|
my problem is that the result appears with the previous question choices and i don't know how to fix it
import java.awt.*;
import java.awt.event.*;
import javax.swing.*;
class OnlineTest extends JFrame implements ActionListener
{
JLabel l;
JRadioButton jb[]=new JRadioButton[5];
JButton b1,b2;
ButtonGroup bg;
int count=0,current=0,x=1,y=1,now=0;
int m[]=new int[10];
OnlineTest(String s)
{
super(s);
l=new JLabel();
add(l);
bg=new ButtonGroup();
for(int i=0;i<5;i++)
{
jb[i]=new JRadioButton();
add(jb[i]);
bg.add(jb[i]);
}
b1=new JButton("Next");
b2=new JButton("Bookmark");
b1.addActionListener(this);
b2.addActionListener(this);
add(b1);add(b2);
set();
l.setBounds(30,40,450,20);
jb[0].setBounds(50,80,600,30);
jb[1].setBounds(50,110,600,30);
jb[2].setBounds(50,140,600,30);
jb[3].setBounds(50,170,600,30);
b1.setBounds(100,240,100,30);
b2.setBounds(270,240,100,30);
setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
setLayout(null);
setLocation(250,100);
setVisible(true);
setSize(600,350);
}
public void actionPerformed(ActionEvent e)
{
if(e.getSource()==b1)
{
if(check())
count=count+1;
current++;
set();
if(current==10)
{
b1.setEnabled(false);
b2.setText("Result");
}
}
if(e.getActionCommand().equals("Bookmark"))
{
JButton bk=new JButton("Bookmark"+x);
bk.setBounds(480,20+30*x,100,30);
add(bk);
bk.addActionListener(this);
m[x]=current;
x++;
current++;
set();
if(current==9)
b2.setText("Result");
setVisible(false);
setVisible(true);
}
for(int i=0,y=1;i<x;i++,y++)
{
if(e.getActionCommand().equals("Bookmark"+y))
{
if(check())
count=count+1;
now=current;
current=m[y];
set();
((JButton)e.getSource()).setEnabled(false);
current=now;
}
}
if(e.getActionCommand().equals("Result"))
{
count=count+1;
current++;
if (count>=6)
l.setText("You have passed the exam " );
else
l.setText( " Repeat the exam after 2 months " );
}
}
void set()
{
jb[4].setSelected(true);
if(current==0)
{
l.setText("Que1: The only time you can cross two solid yellow lines is when ?");
jb[0].setText(" You need to pass");jb[1].setText(" Turning left at an intersection");jb[2].setText(" The car in front of you is driving under the speed limit");jb[3].setText(" You should never cross double solid yellow lines ");
}
if(current==1)
{
l.setText("Que2: It is always illegal to park:");
jb[0].setText(" Three feet of a fire hydrant");jb[1].setText(" At a red curb");jb[2].setText(" In a double parked position ");jb[3].setText(" All of the above");
}
if(current==2)
{
l.setText("Que3: Always leave extra space in front of you when :");
jb[0].setText(" driving in bad weather");jb[1].setText(" When being followed closely");jb[2].setText(" Y ou're following a motorcycle"); jb[3].setText(" all of the above ");
}
if(current==3)
{
l.setText("Que4: When is it ok to drive faster than the posted speed limit?");
jb[0].setText(" It is never ok");jb[1].setText(" When being followed closely ");jb[2].setText(" To keep pace with traffic flow");jb[3].setText(" If you have an emergency ");
}
if(current==4)
{
l.setText("Que5: What does an orange sign mean?");
jb[0].setText(" State highway ahead");jb[1].setText(" Merging lanes ahead ");jb[2].setText(" Construction work ahead ");jb[3].setText(" Divided highway ahead ");
}
if(current==5)
{
l.setText("Que6: If you see a car approaching in your lane you should:");
jb[0].setText(" Slow down");jb[1].setText(" Sound your horn ");jb[2].setText(" flash your lights");jb[3].setText(" All of the above");
}
if(current==6)
{
l.setText("Que7: When driving in heavy fog you should drive with your? ");
jb[0].setText(" Headlights off ");jb[1].setText(" Headlights on low beam ");jb[2].setText(" Parking lights on");
jb[3].setText(" Headlights on high beam");
}
if(current==7)
{
l.setText("Que8: How far ahead should you look when you are on the highway?");
jb[0].setText(" 3 to 5 seconds ");jb[1].setText(" 5 to 10 seconds");jb[2].setText(" 10 to 15 seconds.");
jb[3].setText(" 15 to 20 seconds");
}
if(current==8)
{
l.setText("Que9: What does a flashing yellow light mean?");
jb[0].setText(" Pedestrian crossing ");jb[1].setText(" Proceed with caution ");jb[2].setText(" School crossing ");jb[3].setText(" Non of the above");
}
if(current==9)
{
l.setText("Que10: What does a yellow sign mean?");
jb[0].setText(" State highway ahead ");jb[1].setText(" Construction work ahead");jb[2].setText(" A special situation ahead");
jb[3].setText(" Construction work ahead");
}
l.setBounds(30,40,450,20);
for(int i=0,j=0;i<=90;i+=30,j++)
jb[j].setBounds(50,80+i,200,20);
}
boolean check()
{
if(current==0)
return(jb[1].isSelected());
if(current==1)
return(jb[3].isSelected());
if(current==2)
return(jb[3].isSelected());
if(current==3)
return(jb[0].isSelected());
if(current==4)
return(jb[2].isSelected());
if(current==5)
return(jb[3].isSelected());
if(current==6)
return(jb[1].isSelected());
if(current==7)
return(jb[3].isSelected());
if(current==8)
return(jb[1].isSelected());
if(current==9)
return(jb[2].isSelected());
return false;
}
public static void main(String s[])
{
new OnlineTest("Online Driving Test ");
}
}
|
|
|
|
|
And you expect us to figure out what that is supposed to mean?
Might I suggest you learn how to use arrays or lists properly, rather than repeating code blocks.
|
|
|
|
|
I need code source code for genetic synthesis and particle Swarm optimization in the netbeans environment to simulate cloudsim . Can anyone help me ?
|
|
|
|
|
No. This site does not provide code to order. Try freelancer.com.
|
|
|
|
|
Member 11492407 wrote: Can anyone help me ?
Yes.
Do you have funding in place? I could provide a multi-million dollar project estimate with full requirements.
|
|
|
|
|
|
|
(function(l,f){function m(){var a=e.elements;return"string"==typeof a?a.split(" "):a}function i(a){var b=n[a[o]];b||(b={},h++,a[o]=h,n[h]=b);return b}function p(a,b,c){b||(b=f);if(g)return b.createElement(a);c||(c=i(b));b=c.cache[a]?c.cache[a].cloneNode():r.test(a)?(c.cache[a]=c.createElem(a)).cloneNode():c.createElem(a);return b.canHaveChildren&&!s.test(a)?c.frag.appendChild(b):b}function t(a,b){if(!b.cache)b.cache={},b.createElem=a.createElement,b.createFrag=a.createDocumentFragment,b.frag=b.createFrag();
a.createElement=function(c){return!e.shivMethods?b.createElem(c):p(c,a,b)};a.createDocumentFragment=Function("h,f","return function(){var n=f.cloneNode(),c=n.createElement;h.shivMethods&&("+m().join().replace(/\w+/g,function(a){b.createElem(a);b.frag.createElement(a);return'c("'+a+'")'})+");return n}")(e,b.frag)}function q(a){a||(a=f);var b=i(a);if(e.shivCSS&&!j&&!b.hasCSS){var c,d=a;c=d.createElement("p");d=d.getElementsByTagName("head")[0]||d.documentElement;c.innerHTML="x<style>article,aside,figcaption,figure,footer,header,hgroup,nav,section{display:block}mark{background:#FF0;color:#000}</style>";
c=d.insertBefore(c.lastChild,d.firstChild);b.hasCSS=!!c}g||t(a,b);return a}var k=l.html5||{},s=/^<|^(?:button|map|select|textarea|object|iframe|option|optgroup)$/i,r=/^<|^(?:a|b|button|code|div|fieldset|form|h1|h2|h3|h4|h5|h6|i|iframe|img|input|label|li|link|ol|option|p|param|q|script|select|span|strong|style|table|tbody|td|textarea|tfoot|th|thead|tr|ul)$/i,j,o="_html5shiv",h=0,n={},g;(function(){try{var a=f.createElement("a");a.innerHTML="<xyz></xyz>";j="hidden"in a;var b;if(!(b=1==a.childNodes.length)){f.createElement("a");
var c=f.createDocumentFragment();b="undefined"==typeof c.cloneNode||"undefined"==typeof c.createDocumentFragment||"undefined"==typeof c.createElement}g=b}catch(d){g=j=!0}})();var e={elements:k.elements||"abbr article aside audio bdi canvas data datalist details figcaption figure footer header hgroup mark meter nav output progress section summary time video",shivCSS:!1!==k.shivCSS,supportsUnknownElements:g,shivMethods:!1!==k.shivMethods,type:"default",shivDocument:q,createElement:p,createDocumentFragment:function(a,
b){a||(a=f);if(g)return a.createDocumentFragment();for(var b=b||i(a),c=b.frag.cloneNode(),d=0,e=m(),h=e.length;d<h;d++)c.createElement(e[d]);return c}};l.html5=e;q(f)})(this,document);
|
|
|
|
|
Despite the similar names, Javascript is not the same thing as Java.
What you have there is Javascript - namely, the HTML5 "shiv"[^] which allows older versions of Internet Explorer to use HTML5 features.
Compare the version you've got with the version from GitHub. If they don't match, replace your version with the GitHub version. If you still get a warning, then it's most likely a false-positive.
"These people looked deep within my soul and assigned me a number based on the order in which I joined."
- Homer
|
|
|
|
|
i'm new to Android development. i have downloaded Android Studio. i've made no adjustments by any means to any of the documents. i've attempted to comply with the commands at "developer.android.comtrainingbasicfirstapp" as far as possible (it appears to be for a one-of-a-kind version because there are variations among what it says will be there and what is).
when I try to run (don't forget this is an certainly bathroom-general installation with nothing changed [other than a name for the app, and downloading NDK - again, something not mentioned on the training site]) i am getting to a conversation referred to as "Edit Configuration" (no longer referred to on the schooling web site). placing this up as best i'm able to (there's no "pick deployment goal" as mentioned on the training web page), and urgent run, i'm informed that the configuration is inaccurate but with really no Android indication of the problem. If I select "run besides", i get "Gradle synch failed - repair your assignment and attempt again", again, with no indication of what is wrong.
i've spent numerous hours googling this trouble but with out a fulfillment. some answers consult with different versions, some to conditions in which humans have introduced application code (which has caused the mistake in a formerly operating setup), and a few suggest making very dramatic changes to the files and set up of the development system which i'm loath to do due to the fact, frankly, I do not know what is going on, and do not want to stray similarly and similarly from what have to be an operational default as downloaded.
Can all of us explain exactly what is wrong with the usual, default, project, as provided through Google, what is wanted to run it,and, as an advantage, provide an explanation for why one Earth Google release a development gadget that may not even compile and run the most effective initial software they, themselves, supply?
|
|
|
|
|
I am attempting a course in Java
The instructor wants us to use...
- JDK 8
- JKD 9
- Eclipse
- The Window Builder Editor (if the regular install won't do it)
- JShell
- Android Studio
As best I can tell, those six tools will cover everything I'll need to go through this course.
CURRENT FAILURE: When I go to the URL which the instructor provides for placing JDK 9 on my machine, I arrive at a screen which has no link nor any observed instructions/directions/procedure/whatever for installing that version on my machine.
My machine is Windows 7
Advice, Suggestions, and Directions are welcome.
|
|
|
|
|
Use a web search to find the JDK 9 download page: Java SE Development Kit 9 - Downloads[^].
If you are using a 32-bit Windows 7, you are out of luck:
JDK 9 is only available for 64-bit operating systems.
|
|
|
|
|
Jochen Arndt wrote: If you are using a 32-bit Windows 7
Good thinking. I'm pretty sure that this is a 64 bit machine, but I don't know the magic sequence of where to click to confirm/refute this.
I just spent five minutes clicking around on likely places, but couldn't find it.
Alms for the Clueless ?
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
