|
Hmmmm,
Many years ago I was tasked with writing a VDR[^] for the IMO[^] and the Det Norske Veritas[^] and most of the signals were coming in at around 10Hz and I was required to log them in 'real time'. Back then the IEEE defined 'real time' as 1.5 times the signal rate.
For performance reasons I chose SQLlite[^] and a single BLOB[^] in the voyage data recorder.
Worked great for a few years. Then we had two incidents, "incidents" in the maritime industry usually means millions of dollars of damage or greater. No problem I thought, let's pull the black box and see what happened!
The BLOB fields did not match the C++ structs we were using for logging the signals, something was off. After a lengthy forensics analysis we realized that the office in Norway was using a different build and the Norwegian C++ structs did not match with the U.S. code. They were somehow compiling the vessel navigation software with different signal headers.
The 'BLOB' field did not give us any evidence of what was different and we wasted an enormous amount of time investigating this. Also... these millions of dollars of damage were disputed between multiple nation states. In the maritime industry these judgements are decided by arbitrators rather than an actual legal system.
[cheers]
modified 28-Aug-21 5:48am.
|
|
|
|
|
Randor wrote: I should probably delete this so...
If you do, I will be glad I read it before that happens! Your stories are always interesting.
|
|
|
|
|
I have worked with an organisation who catered to the vagaries of traders, who made them lots of money, the front end software allowed basically anything to be entered into a value field (many of them). The software sorted out how to deal with the garbage that was entered, the DB had no rules or definitions everything was a string (except dates, even they were not that stupid). So you would see 5.2m 300k 1.12b 3.3mill etc. Every time the system broke because some idjit entered new garbage the software company made $1000s to adjust the UI.
I was tasked with generating reports from such rubbish. Your premise sucks big grey donkey balls! Am I bitter and twisted after dealing with such garbage in, dammed right I am. I designed data structures that worked, were fast, efficient and anyone could use the data without knowing any arcane rules living in the UI.
Never underestimate the power of human stupidity -
RAH
I'm old. I know stuff - JSOP
|
|
|
|
|
I can just imagine what was entered by traders who blew up their accounts. 
|
|
|
|
|
5teveH wrote: Including 'DB rules' in the code/UI. First of all, we already do this. I'm pretty sure most developers would define the MaxLength for a TextBox and use a Calendar Widget for date input. Secondly, validation logic should be developed as reusable code - which minimises the need for future developers to learn and re-code that logic.
Called SQL. The "S" standing for standard, and SQL92 still being supported.
5teveH wrote: Performance. You are going to have to take my word for it, but I am 100% sure that with less disk, less CPU and less memory, I can deliver better performance than could be achieved using a traditional RDBMS.
I'll outpace you with a traditional RDBMS. I will take that bet for a banana (not allowed to bet more, but I owe over a trillion bananas now).
RDBMS is not optimized for speed, but rationality. It is an abstraction layer (what lots of us pretend to write) that abstracts away physicical storage. You no longer have to remember at what location a record begins, because the RDBMS handles that. The RDBMS was for years our Data Abstraction Layer (The DAL, which many a company asked money for, while doing one on one calls).
5teveH wrote: Also, indexing data does not have a performance hit
"Almost" none; but there's a nice gain on retrieving the index. That's why indexes exist.
5teveH wrote: And the greater the volume of data, the more confident I would be that performance would be better.
I'd be looking mostly at the capabilities of the dba.
Also; we restrict some things at db level because that's how it is done. Given a system, where some desktop and some phone app interact with a db, where do you put the restriction? Come on, we been doing this for 30 years 
You don't need to use an RDMS, nor need to understand what BNF is. That's your choice
Bastard Programmer from Hell 
"If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.
|
|
|
|
|
5teveH wrote: Also, indexing data does not have a performance hit.
This is flat out wrong. Indexing unstructured data takes a lot of CPU time and disk time. Take a look at the amount of CPU and disk time spent by full text index systems during data insertion and updating. Updating can actually be worse as it requires you delete from the index as well as add to it.
Indexing fixed size fields is still expensive, but the disk IO is easily an order of magnitude less, which translates to better performance.
|
|
|
|
|
obermd wrote: This is flat out wrong.
All I can say is that, after nearly 40 years of working with this database, (Pick/Universe), I have never once seen a performance hit from indexing variable length data. So I'm still backing myself on this one.
|
|
|
|
|
I worked with Pick long ago and my experiences were not very good. My problems with Pick at that time were not its performance btw. but the fact that it was a closed system, OS and database combined.
We ran into serious trouble when the database got corrupted, there were no tools available like we have for DOS or Windows systems and some customers lost their data.
I can only hope that new versions of Pick have been improved in this respect.
|
|
|
|
|
These days I work on Universe - a Pick derivative, which runs on Unix & Windows. It definitely was a very closed system, back in the day, but it's a whole lot more open these days.
|
|
|
|
|
Then you must be a Master of the Universe 
|
|
|
|
|
 if only
|
|
|
|
|
5teveH wrote: I thought this might stir up a few of the 'traditionalists' on here! 
I love this.
I think my post was one of the few that wasn't upvoted because I partially agreed with you
I'd never heard of Pick.
Doesn't seem like something I'd want to use.
The Wiki page has some interesting facts about it though...
"Pick was originally implemented as the Generalized Information Retrieval Language System (GIRLS) on an IBM System/360 in 1965" (who said developers don't get the girls!? )
"It is named after one of its developers, Richard A. (Dick) Pick." (seriously, Dick Pick!? )
On a serious note, Pick seems to pre-date SQL and before SQL the only thing you had was NoSQL.
Then after some thirty years of everything SQL, NoSQL gained popularity again.
NoSQL didn't come back because SQL was so good at everything and speed and scalability were two of the main bottlenecks.
Updating a SQL database schema is also generally a hassle, with large tables taking hours to update.
No such thing in schemaless databases, just update the software and you're good to go!
It's fun that we're all using "schemaless" REST services and no one bats an eye, but do the same for a database and everyone loses their minds.
That said, SQL adapted and NewSQL databases now combine some of the speed and flexibility of NoSQL with the safety of SQL.
I've found people are very defensive of SQL and very afraid of NoSQL (to the point of people becoming insulting and personal when someone, mostly me, mentions NoSQL).
People don't like to get out of their comfort zones and they'll fight to stay there 
|
|
|
|
|
Yes, it predates SQL - and back in the day, there was nothing like it. The original Pick database was actually a complete system. It ran native on hardware - no O/S. It came with it's own query language, procedural language and programming language. And, as a result, it was a very closed system. But I think one of the main reasons the Pick database didn't get the traction it deserved was Dick Pick's licencing 'antics'. It resulted in a whole bunch of Pick 'franchisees' - each with their own implementation and responsibility for maintenance and upgrades. All competing against each other - and against Pick Systems! If Pick Systems had had a savvy business strategy and half-decent marketing, there would probably be no such thing as SQL.
These days I work on Universe - a Pick derivative, which runs on Windows and Unix. It's nowhere near as closed off as those original systems, but definitely still has it's limitations. For instance: unlike most NoSQL databases, it doesn't cluster.
Sander Rossel wrote: I've found people are very defensive of SQL and very afraid of NoSQL (to the point of people becoming insulting and personal when someone, mostly me, mentions NoSQL). People don't like to get out of their comfort zones and they'll fight to stay there
Yep! That's why I got my insults in first - cheekily, calling them 'traditionalists'. But, to be honest, I knew it was going to be a 'hard sell'. When I was first told about how the Pick database worked, (and there's a lot of other left-field stuff that I haven't mentioned), I was equally sceptical. I suspect no one is going to, properly, get NoSQL until they've worked with it - and actually done the job properly.
|
|
|
|
|
|
You will not outperform an RDMS since you will have to reproduce the lower-level code where all the performance is.
The query optimizer is not perfect; that is the only place where you can outperform; depending on the query and only if you appreciate granularity.
As far as textbox length goes, it's part of defensive programming. Someone will always try to insert 5,000 spaces and one character, just to see if they can.
It was only in wine that he laid down no limit for himself, but he did not allow himself to be confused by it.
― Confucian Analects: Rules of Confucius about his food
|
|
|
|
|
As far as NO SQL databases go, I think it is just another one of those cyclic “fads” like centralized vs distributed, rent vs own, agile vs waterfall, etc.
We used IBM Lotus Notes for every little one off app+DB that we needed.
Then we ditched Notes for email, so a lot of the more critical apps went to custom SharePoint workflows, now those are all being converted to Low Code/No Code apps on a cloud platform, in 10 years when the cloud platforms start leveraging their “hostage” power, everyone will shift back to on premise.
In 10 years, I suspect there will be some hybrid solutions where you have 2 refrigerator sized appliances that can support a 100,000 person enterprise. You just replace one of the refrigerators every few years to stay on maintenance.
|
|
|
|
|
I had my students develop the DB first; from that, one can easily build a web or desktop UI, independent from each other. Even other apps can access it, and when using a mainstream DB this decouples the data-layer from the rest.
5teveH wrote: First of all, we already do this. I'm pretty sure most developers would define the MaxLength for a TextBox and use a Calendar Widget for date input. Secondly, validation logic should be developed as reusable code - which minimises the need for future developers to learn and re-code that logic.
The max length, as a simple example, does apply as much to the DB as the UI. Most generate their UI based on the restrictions provided by reflection.
5teveH wrote: Performance. You are going to have to take my word for it, but I am 100% sure that with less disk, less CPU and less memory, I can deliver better performance than could be achieved using a traditional RDBMS. Also, indexing data does not have a performance hit. And the greater the volume of data, the more confident I would be that performance would be better.
You don't have to take mine; you gimme a model, I put it into 3BCNF and hand you the metrics. I do not ask for belief, I provide measurements.
Indexing data HAS a performance hit; the PC has to do something additional, how can you claim that the extra processing has no performance hit??
Bastard Programmer from Hell 
"If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.
|
|
|
|
|
Eddy Vluggen wrote: Indexing data HAS a performance hit
True. I was responding to a comment about indexing variable length data having a much bigger performance hit - and that was not at all clear. Yes, I totally accept that everything you do on a system is going to take some resource - including indexing data. But, certainly on the database I work with, (which is completely variable length), there is no 'real world' impact when you index data. Which, I don't think is surprising. If you design a database from the ground up, to be variable length, you are going to deal with things like indexing variable length data as the rule, rather than the exception.
|
|
|
|
|
Variable length is a pointer to a blob. Indexing gives a performance gain if done right.
Enjoy. I'll be asking double to clean up after you.
Bastard Programmer from Hell
"If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.
|
|
|
|
|
You are going to have to take my word for it, but I am 100% sure that with less disk, less CPU and less memory, I can deliver better performance than could be achieved using a traditional RDBMS.
I'm not gonna. If you were such a wizard, someone would have hired you to write the next PostgreSQL and make a mint.
|
|
|
|
|
5teveH wrote: You are going to have to take my word for it, but I am 100% sure that with less disk, less CPU and less memory, I can deliver better performance than could be achieved using a traditional RDBMS. Also, indexing data does not have a performance hit. And the greater the volume of data, the more confident I would be that performance would be better.
And from a different thread...
5teveH wrote: after nearly 40 years of working with this database, (Pick/Universe),
I have been dealing with database for 40 years. Different databases. Different industries. Different types of enterprise systems.
And during that time I have also seen databases change. For instance just the infrastructure that they run on has changed enough that things that used to matter do not and things that were never even looked at before matter much more now. So that 40 years of experience doesn't even translate well into deciding what one should do now versus one did then. About the only real thing it allows is telling stories and being able better to recognize 'tribal knowledge' which is based on something that is no longer valid.
The first comment it completely open ended without restrictions and not even providing specific definitions.
In my experience making broad sweeping claims about anything always leads to one thing - failure. Followed by a lot of rationalizations and back sliding about what was really meant by the original claims.
For starters "performance" can mean almost anything but in the real world customers have real needs for what "performance" means. They don't care about benchmarks. They do care about how long they have to sit around waiting for results to show up on the screen and how long it takes to come up with new features that they think they need. (The two are not complementary.)
Moreover in terms of actual performance and the enterprise level performance is achieved by requirements modifications and not technology. One might gain a 1% boost with technology but might gain 6,000% by changing requirements. That last is based on a real world example.
|
|
|
|
|
Is USB just a pollinating insect that comes from America?
"I have no idea what I did, but I'm taking full credit for it." - ThisOldTony
"Common sense is so rare these days, it should be classified as a super power" - Random T-shirt
AntiTwitter: @DalekDave is now a follower!
|
|
|
|
|
USB kidding me
The less you need, the more you have.
Even a blind squirrel gets a nut...occasionally.
JaxCoder.com
|
|
|
|
|
Honey, I hope you queen see how silly this question is.
|
|
|
|
|
It's not often that one of these make me think, just usually chuckle, but this one did both.
"When you are dead, you won't even know that you are dead. It's a pain only felt by others; same thing when you are stupid."
Ignorant - An individual without knowledge, but is willing to learn.
Stupid - An individual without knowledge and is incapable of learning.
Idiot - An individual without knowledge and allows social media to do the thinking for them.
modified 19-Nov-21 21:01pm.
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
