Re: Which approach is better for this problem? - C# Discussion Boards

SledgeHammer0110-Feb-12 3:52

10-Feb-12 3:52

Pete O'Hanlon wrote:
I didn't see anything in the OP that suggests that he has any need for that type
of scale.

Yes, OP confessed he only has a need for about 5 clients. Carving stone tablets and shipping them via carrier pigeon would work at that scale.

Pete O'Hanlon wrote:
Well, we have one app which is currently handling about 5K clients (granted not
push), using TCP. I didn't see anything in the OP that suggests that he has any
need for that type of scale.

Cool. Yes, that is what I've seen on google as well. Something in the 3k to 5k range.

If you want something that'll do 10k+ clients bi-directional though, as far as I know, only TCP/IP w/ IOCP can handle that.

Yeah, but 5 != 10,000 Smile | :)

jschell9-Feb-12 13:34

9-Feb-12 13:34

SledgeHammer01 wrote:
I'd DEFINITELY avoid WCF because its very slow and network traffic is very large
(relative to the real data) because its all SOAP & WSDL.

There is overhead regardless of the transfer mechanism.
And if there is actual concern about the traffic size and message throughput then a wide range of conderations come into play. And if that is really a concern then myself I would go with import files and moving them via file transfer mechanism.

But excluding exremely high volume systems on a modern data center setup any WCF implementation is going to be sufficient.

SledgeHammer019-Feb-12 15:17

9-Feb-12 15:17

I've never tested it on WCF myself, but I did a quick google search and ONE guy claimed he wrote a WCF service that could handle 7000 "concurrent clients". Most other links I found are claiming closer to around 3000 to 5000 for a "real" service. The 7000 client guy had a very light weight sample.

If you have < 100 clients or whatever, pretty much anything will work. Maybe even at 1000 clients anything will work. Even if its all the way up to 5000 on a real service, you do realize that is very few clients in the internet scheme of things, right?

I did ask the guy how many clients he had...

But regardless of that fact... WCF doesn't support push.

Pete O'Hanlon9-Feb-12 21:59

Pete O'Hanlon

9-Feb-12 21:59

SledgeHammer01 wrote:
But regardless of that fact... WCF doesn't support push.

So, you can't do any form of push with WCF at all[^]?

*pre-emptive celebratory nipple tassle jiggle* - Sean Ewington

"Mind bleach! Send me mind bleach!" - Nagy Vilmos

My blog | My articles | MoXAML PowerToys | Mole 2010 - debugging made easier - my favourite utility

SledgeHammer0110-Feb-12 3:48

10-Feb-12 3:48

Ah, didn't know you could do that Smile | :)

Pete O'Hanlon10-Feb-12 3:56

Pete O'Hanlon

10-Feb-12 3:56

It is pretty cool.

*pre-emptive celebratory nipple tassle jiggle* - Sean Ewington

"Mind bleach! Send me mind bleach!" - Nagy Vilmos

My blog | My articles | MoXAML PowerToys | Mole 2010 - debugging made easier - my favourite utility

jschell10-Feb-12 9:20

10-Feb-12 9:20

SledgeHammer01 wrote:
Even if its all the way up to 5000 on a real service, you do realize that is very few clients in the internet scheme of things, right?

Not sure what you mean.

However a site that has 5000 simultaneous clients would be a site with very high traffic.

And if the OP wants to do a machine to machine database transfer that involves 5000 simultaneous clients then I suspect a flawed design and/or flawed architecture. Not to mention a flawed question.

SledgeHammer01 wrote:
WCF doesn't support push.

And that matters why? Do you think that most client/server applications should be designed with push? Do you think that serves the OPs goal?

SledgeHammer0110-Feb-12 9:34

10-Feb-12 9:34

jschell wrote:
However a site that has 5000 simultaneous clients would be a site with very high
traffic.

Well, CodeProject has 8.4M members... throw out the spammers, etc and consider that it is a niche site.... according to the stats there are 38k users online at this typing.

Games will have more then 5000 simultaneous clients easily.

Anyways, this is all a moot point since OP says he will have only 5 Smile | :)

jschell wrote:
And that matters why? Do you think that most client/server applications should
be designed with push? Do you think that serves the OPs goal?

I stand corrected on that it doesn't support push as Pete posted an article showing callbacks.

But yes, the OP would be better off with a push design vs. polling every second which was his original issue.

He was polling the database every second and complaining of jerky refreshes. A push vs. poll design solves that problem.

jschell12-Feb-12 7:29

12-Feb-12 7:29

SledgeHammer01 wrote:
Well, CodeProject has 8.4M members... throw out the spammers, etc and consider that it is a niche site.... according to the stats there are 38k users online at this typing.

First this site uses http, so the fact that the is 38k sessions has nothing to do with how many connections there are at one time.

Second I am sure google and facebook have even more.

However extrapolating that every or even most websites will have that much traffic would easily exceed the population of the planet. So most web sites will not experience that much traffic. Ever.

SledgeHammer01 wrote:
He was polling the database every second and complaining of jerky refreshes. A push vs. poll design solves that problem.

I can't find that post. And if it was made...
1. It is a GUI problem not a data transport problem.
2. Humans can't react in one second so the requirement is pointless.
3. All databases that I know of can easily handle polling with 5 clients with a one second poll rate.

SledgeHammer0112-Feb-12 8:43

12-Feb-12 8:43

jschell wrote:
However extrapolating that every or even most websites will have that much
traffic would easily exceed the population of the planet. So most web sites will
not experience that much traffic. Ever.

Sounds like you really need to get out into the real world if you think 5000 simultaneous connections is a lot. Game servers have WAY more. Brokerages that offer real-time stock tickers have WAY more. Those are some REAL WORLD examples for you since I know you have a really difficult / impossible time wrapping your brain around "hypothetical concepts" Big Grin | :-D

Anyways, that's not the point. The OP clearly stated he would have 5 low bandwidth connections later on, so he doesn't need a high bandwidth, scaling solution.

jschell wrote:
I can't find that post. And if it was made...

That was his first post.

jschell wrote:
1. It is a GUI problem not a data transport problem.

False. Again. ADO.NET out of the box closes the connection after every query / stored proc call and uses TDS over TCP/IP. Open + Connect + Execute Query + Merge results + close connection is actually a relatively expensive call. If you are doing this in a background thread, it wouldn't make the GUI jerky per say, but it would affect performance. If you are doing it in the foreground thread, thats a GUI issue. However, you can't update the GUI from a background thread directly (without invoke), so that'll affect the GUI performance.

jschell wrote:
2. Humans can't react in one second so the requirement is pointless.

You just don't like real requirements do you? Big Grin | :-D

Reading your posts and playing arm chair psychiatrist, I think I have finally figured out your problem. You have spent too many years implementing "business requirements" at business oriented companies that don't care about performance. Thats cool. I work at a company right now where the boss doesn't care about performance. He doesn't care if it takes 1 minute to process a widget vs. 1 second. I mean, sure he is completely disregarding the fact that we have to process BILLIONS of widgets and that 1ms per widget is 277 HOURS per billion widgets, but thats neither here nor there.

As indicated above, game servers & stock tickers are REAL WORLD examples of stuff that needs to happen quickly.

Is that relevant to the OP? No, not really.

jschell wrote:
All databases that I know of can easily handle polling with 5 clients with a one
second poll rate.

How weird. My boss built a service that polls a hefty Windows 2008 + SQL 2008 box on a single connection / single thread and the box can barely keep up.

jschell14-Feb-12 13:27

14-Feb-12 13:27

SledgeHammer01 wrote:
Sounds like you really need to get out into the real world if you think 5000
simultaneous connections is a lot. Game servers have WAY more. Brokerages that
offer real-time stock tickers have WAY more. Those are some REAL WORLD examples
for you since I know you have a really difficult / impossible time wrapping your
brain around "hypothetical concepts"

And what is the percentage of websites that generate those numbers versus the total number of websites in the world?

SledgeHammer01 wrote:
That was his first post.

First post at this time says nothing at all about "jerky refreshes".

SledgeHammer01 wrote:
False. Again. ADO.NET out of the box closes ...

The phrase "jerky refreshes" suggests a GUI display problem to me.

SledgeHammer01 wrote:
You just don't like real requirements do you? Reading your posts and
playing arm chair psychiatrist, I think I have finally figured out your problem.
You have spent too many years implementing "business requirements" at business
oriented companies that don't care about performance.

Current server that I wrote and support has a measured rate based on realistic traffic of 100 TPS (Transactions Per Second) for a single box, with a burst rate of 600 over 3 seconds. The expected volume for the largest servable market business segment for that market is 2000 TPS seconds which translates to 20 boxes. That server currently runs in two geographical distinct redundant data centers supporting remote and local and remote failover.

So I would say that your 'guess' is wrong.

SledgeHammer01 wrote:
My boss built a service that polls a hefty Windows 2008 + SQL 2008 box on a
single connection / single thread and the box can barely keep up.

I worked at a company where the VP would write SQL queries that would shut down the database and 200+ call center workers in the process. But then the queries were poorly written.

SledgeHammer0114-Feb-12 13:58

14-Feb-12 13:58

jschell wrote:
And what is the percentage of websites that generate those numbers versus the
total number of websites in the world?

Who cares? You said 5000 simultaneous connections is alot and I said it wasn't.

jschell wrote:
Current server that I wrote and support has a measured rate based on realistic
traffic of 100 TPS (Transactions Per Second) for a single box, with a burst rate
of 600 over 3 seconds. The expected volume for the largest servable market
business segment for that market is 2000 TPS seconds which translates to 20
boxes. That server currently runs in two geographical distinct redundant data
centers supporting remote and local and remote failover.

So I would
say that your 'guess' is wrong.

No, it sounds about right. You need 20 boxes to handle 2000 (TPS) simultaneous connections??? WTF??? Thats horrible dude.

If you implemented a game server that had avg traffic of 1M users online at any given time you would need, what, 10,000 boxes? LMAO.

Sounds like YOU should have used C++ Winsock TCP/IP with I/O completion ports. Could have hit your 2000 TPS rate on 1 box.

jschell15-Feb-12 8:36

15-Feb-12 8:36

SledgeHammer01 wrote:
Who cares? You said 5000 simultaneous connections is alot and I said it
wasn't.

I stand by what I said.

Most sites, even ones with significant traffic, will NOT have that many simultaneous connections.

SledgeHammer01 wrote:
No, it sounds about right. You need 20 boxes to handle 2000 (TPS) simultaneous
connections??? WTF??? Thats horrible dude

Sigh...That maximum volume represents 1 trillion dollars in yearly volume.

Pretty sure that most businesses would be more than happy with 20 servers at that rate.

SledgeHammer01 wrote:
If you implemented a game server that had avg traffic of 1M users online at any
given time you would need, what, 10,000 boxes? LMAO.

Do you know what simulteneous connections mean?
Do you understand that a connection is not the only processing task that a server must implement in regards to actually doing something with a connection?

You know that google has with low end estimates, 200,000 servers? Other estimates put that even higher.

So exactly what are your personal credentials in terms of high volume delivered running products in the commercial realm?

SledgeHammer01 wrote:
Sounds like YOU should have used C++ Winsock TCP/IP with I/O completion ports.
Could have hit your 2000 TPS rate on 1 box.

I suspect that if I wanted to replicate the World of Warcraft with no other business requirements I probably would have made that technical choice. But since the business requirements of what I am delivering, which isn't WoW, is met completely and sufficiently by the current model.

And the vast majority of real, not imaginary, businesses would also have their real needs met in the same way.

BobJanova9-Feb-12 5:05

BobJanova

9-Feb-12 5:05

It depends on exactly what you want to do, but on the basis that this is actually an application type requirement and not just database duplication, you should probably write a client and a server application (using sockets) and define a suitable protocol for requesting and sending messages. That way you're not tying yourself to a particular database provider or to the architecture of the old system (too badly).

AmbiguousName9-Feb-12 7:03

9-Feb-12 7:03

Yes this is actually application requirement and not just data duplication.

Dan Mos9-Feb-12 5:16

9-Feb-12 5:16

My post will assume that clients and server are under the same domain/firewall.
If that's not the case, please ignore it.

OK, so it most definitely depends.

IF there are lots of clients(100+) and lots of Hits to SQL Server(or whatever) like a Tracing Application(LOts, batchs, labels...):
I would go for a replication solution. A local DB on each client(SQL express or ...) the sql server and a socket based communication
between clients and server. Every 10 minutes or so the local machine would send the commands to the Soket server and in turn
the "socket server" would run some batch inserts on the sql server sending back to the client the result(succes or failure).
You'll also need to set up a Replication app/UI.
It's a complex solution but for let's say 100+ clients each sending a insert every 10-20 seconds it's better then opening a connection each time.

ELSE a direct ADO.Net solution I think would be best.

All the best,

Dan

AmbiguousName9-Feb-12 7:02

9-Feb-12 7:02

- This application is definitely on the same domain (most probably on LAN).
- Number of clients will be no more than 5.

MDL=>Moshu wrote:
ELSE a direct ADO.Net solution I think would be best.

On the above info, can I say that you are inclined to direct access to sql server (remotely), on machine 1?

Dan Mos9-Feb-12 7:13

9-Feb-12 7:13

Hold On.

What I understood(wrongly) is that machine 2 is server the rest are clients.

Does machine one needs to send data to server and server also needs to send data to machine 1(clients)?

If machine 2 needs to send data to machine one too then:

I would opt for a simple socket architecture to send data from machine 2(server) to machine 1(clients)
and regular ADO.Net for the other way around.

I misunderstood your requirements. My bad.

All the best,

Dan

AmbiguousName9-Feb-12 7:30

9-Feb-12 7:30

What you understood wrongly (by your perception) is actually right. Smile | :)

Machine 2 is server and it will not send data (of any sort) back to machine 1. It will just get the data and store it in the sql server. Machine 1 (the client) is going to send this data.

I hope im not confusing you.

Dan Mos9-Feb-12 7:42

9-Feb-12 7:42

No, not at all.

Then ado.net.

No remote needed.

String con = "connstring for machine 2";
....

And just send data from client to server using standard insterts/queries statements.
Smile | :)

All the best,

Dan

AmbiguousName9-Feb-12 7:48

9-Feb-12 7:48

thnx for your valuable input Smile | :)

Dan Mos9-Feb-12 8:07

9-Feb-12 8:07

glad I could help Smile | :)

All the best,

Dan

SledgeHammer019-Feb-12 15:25

9-Feb-12 15:25

overloaded Name wrote:
This application is definitely on the same domain (most probably on LAN).
-
Number of clients will be no more than 5.

Oh... then who cares? pretty much anything can handle 5 clients. I'm surprised you got jerky refreshes with a polling implementation with only 5 clients.