|
Hi,
I need to write a High Performance TCP/IP server for receiving the data from the GPS devices and logging it to the database for display on Google Maps. There will be thousands of GPS devices connected to the GPS server and will be transmitting the data every 30 seconds and the display will be set to refresh typically every minute. I am looking for the help in specific areas, I will write the code myself and believe me I will post the complete developed and tested code on this same forum. I understand the following challenges
1.Concurrent TCP/IP connections
2.Database table bottlenecks
I would like to go for one table per device groups and different ports for devices with different protocols. I am assuming that I will need to write the following components.
1.Data Receiver
2.Parser as per the protocol and the commands
3.Data Cache
4.Database writer
Receive the bytes, log the received commands to the cache, implement the lazy writer to flush the data every minute to the database using different thread. Please comments on the following.
1.Does the GPS device disconnects after sending one command or it keep the connection to send the data further like long polling?
2.Is it better idea to maintain the thread pool and keep one thread per device? How many devices can be connected simultaneously to one port?
3.If I assume that device does not disconnect the client after transmitting the bytes to the server, should I keep the channel alive on the server to keep on reading the data every 20 seconds or whenever available?
4.Is it fair enough to have the separate threads for updating the data to the database server? Say data retrieval every 20 seconds and making it persistent to the database every one minute?
5.Is there any other better protocol rather that raw TCP/IP? I guess the other protocols like http etc should be slower as the header size itself is very big.
6.What is better choice Synchronous or Asynchronous sockets? Is it feasible to use UDP?
7.What is better choice? C# and SQL Server or JAVA and mySQL on Linux?
Many Thanks
Vipin
|
|
|
|
|
Not an answer but a few comments.
- No one can answer that question; you need to get the documentation for the device to see how it operates.
- Any number of devices (up to system saturation point) can connect via the same port, in the same way that all HTTP requests go to port 80.
- Depends on how the device operates.
- Possibly, but performance tests are needed to be sure.
- TCP or UDP are probably best choice of protocol.
- Again, testing will show which is best. See also point 5.
- C# or C++ and SQL (or other) server. I would certainly not use Java for such a performance intensive program, and you need a fast database.
|
|
|
|
|
Writing a high performance tcp server is very tricky.
You have to have a good understanding of overlapped IO. Managing kernel receive buffers, and a huge mess of stuff that's not obvious from simple 'my first tcp server' type articles on the internet
Strongly suggest you use professional messaging wrapper around tcp
Eg; ZeroMQ is something that hides a lot of these details from you
http://zeromq.org/[^]
Another possible recommendation is WCF with the tcp binding
I would not write your own tcp server just for the hell of it; it will no doubt be terrible compared to the above products
Matt.
|
|
|
|
|
Thanks Matt,
I also thought of using WCF tcp binding but the WCF may be much slower as compare to raw sockets however dealing with raw sockets may be more tricky and challenging but still possible to implement.
Thanks
Vipin
|
|
|
|
|
|
|
Unless you have 5+ years experience writing high performance TCP messaging servers
Your code is going to be way, way, worse than what's in WCF and not as scalable.
Unless you know how TCP buffers are managed by the kernel; why it's bad to use thread-per request; what overlapped IO is and what an NT fiber is
Then you're 0% the way qualified to write your own TCP server and shouldn't even attempt it.
Writing a high performance 'good' tcp server implementation is 1 to 5 years development effort on its own.
You can write a raw sockets server using a thread-per-request model in like a few hours Sure; But you'll start running into issues with many clients pretty rapidly.
If you want to reduce the packet size of WCF. Use google protobuff binding for WCF. It makes insanely tiny packets.
There's really no need to cause some other poor developer to work on a naive server implementation when there are very fast and very scaleable tools (WCF, ZeroMQ) that take care of all of this for you
Writing a custom tcp server is pretty much like going "I'd like to write my own kernel scheduler"
It's easy to get going, takes a lifetime to do it right though. That's why ZeroMQ and WCF exist.
|
|
|
|
|
Thanks Matty
If we use the WCF, i guess it would not be able to communicate with the RAW tcp client, we will need to write some code for WCF layer to make it communicating with the RAW tcp connections. Thanks for suggesting the ZeroMQ, i will explore these libraries and will let you know.
|
|
|
|
|
|
You seem to concentrate on the data collection side only.
You also said: "the display will be set to refresh typically every minute". Ehm, where? How many "display" clients? Webbrowser? And which data are to be displayed? The current location of thousands of devices?? The path taken by a selected device during the last timespan? This involves querying data from the database, and perhaps also some calculations on the data in the database. Or additionally some calculations on the data received from the database in an application...
|
|
|
|
|
Thanks Bernhard
The display will be the browser and typically one user for every 30 devices. Which most of the time will display the last stats of the device. I think we can store the last positions and other states in a separate table. Or some mechanism can be developed that can provide the data of current stats to the browser without hitting the database. The devices will be 2000 and will be transmitting the data every 20 seconds. On a maximum there can be 60000,000,00 records in the transaction table. Which SQL Sever can manager easily without loosing the performance.
|
|
|
|
|
Interesting question.
Firstly, I think you need to be sure on how each GPS would connect. You suggest TCP, you need to get this confirmed and you need to find out whether it connects, delivers its information then disconnects (likely) or it leaves the connection open (unlikely).
How many GPS units are you talking about, how many thousand? If it's a thousand, and they update twice a minute you're only looking at 33 connections/second which I don't find in any way scary. From what I know about GPSs (which isn't a great deal) they normally send a very small packet just containing location information.
Then I'd implement a producer-consumer queue. Each incoming connection takes the data and just sticks it on the queue and disconnects. On the other end you have something (or more than one) which pulls these off the queue and writes them to the database.
I'd go for asynchronous constructs (BeginAccept, BeginReceive) on the reception end as these won't require a long running thread and scale well.
Regards,
Rob Philpott.
|
|
|
|
|
how to make slave monitor show something same as primary monitor? thx
|
|
|
|
|
Easiest way is to set it in your display properties: Right click the desktop, select Screen Resolution, and set "Multiple Displays" to "Duplicate these displays"
Or, use WIN+P then select "Duplicate"
Can you do this from C#? Dunno - you may be able to via the registry, but that would need an elevate application which means kicking the UAE - so your user has to react to that anyway.
What exactly are you trying to do?
Those who fail to learn history are doomed to repeat it. --- George Santayana (December 16, 1863 – September 26, 1952)
Those who fail to clear history are doomed to explain it. --- OriginalGriff (February 24, 1959 – ∞)
|
|
|
|
|
Thank your replay.My os is xp, that does not suppport to set "Multiple Displays" to "Duplicate these displays", it just support extend primary monitor to slave.
I need to use double monitors in project, and make slave monitor show want i do for customer in sometimes,so i want to do this with c#
|
|
|
|
|
On my XP laptop the duplicate display functionality is available only through the graphics adapter manufacturer's own control panel (nvidia). I do find it's of limited use unless the resolution and geometry of both displays are the same.
For example if I attach a 1024x768 external monitor then the laptop's 1400x1050 display will only use the central 1024x768 portion.
Alan.
|
|
|
|
|
how to create a background worker function that run to generate a long-running report on a diferent thread?
|
|
|
|
|
|
Rather than using a BackgroundWorker, try using the Task Parallel Library.
|
|
|
|
|
I am trying to trace IP-Address who is deleting files from my share folders. I have write a code to get information when trying to do something in my share folder but, I can't get the IP from where the delete event's is occurred.
Yes.......
|
|
|
|
|
Yep. Windows doesn't track where the command came from.
This sounds more like someone shouldn't have permissions to your shared folder. Did you setup security on the share or did you just let "Everyone" have Full Control permissions over the folder?
|
|
|
|
|
If you know the shared folders, you would be much better off restricting who has access to the share.
|
|
|
|
|
In the below code sample, a base class constructor was called, with an object created in the argument list. I need to dispose the object created in the argument list. Sample code below
//base class constructor
public Class1(String str, Dbcontext dbcontext)
{
this.str = str;
this.dbcontext = dbcontext;
}
...
//derived class
public Class2(String str, Dbcontext dbcontext):base("xxx", new YYYEntities)
{
}
Please share your thoughts to dispose the highlighted object created.
Adam
|
|
|
|
|
The GC will do it.
Any particular reasons why you're asking, or just curious?
Bastard Programmer from Hell 
If you can't read my code, try converting it here[^]
|
|
|
|
|
Not your job. If you didn't create it, don't destroy it. If you didn't open it, don't close it. If you didn't set it, don't clear it.
You'll never get very far if all you do is follow instructions.
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
