|
Well, of course joining two tables will take some time. It doesn't matter where you join them -- in the DB or in the Python code --- it's going to take some time.
However, it is always better to let the DB do what it is good it --- working with data.
I'm always amazed the number of developers who don't recognize that simple divide:
- EVERYTHING related to data (querying, filtering, selecting) should be done in the Database.
- NOTHING else should be.
Truth,
James
|
|
|
|
|
Sounds like you have a combination of emotional maturity and political savvy.
I mean, you could wrote a few queries that prove him wrong. And given 10 minutes, you could explain the data structures underlying a relational database in sufficient detail to show when joins will be expensive, and when they won't.
Often enough, though, being right (and proving it) hurts you. Definitely better to know when to drop it and move on. 
|
|
|
|
|
Marc,
Column counts are a huge issue. I can't stand "Select *" for anything more than debugging!
I was called in to help fix an Oracle Speed Issue. The result was a query that brought back 10,000 times less data! They were doing "Select *" on a 200 column table, joined with 2 other tables. Where they needed 12 columns. And they were looking at the "full table scans".
(imagine 12 columns that fit on a 80 character wide screen. Imagine retrieving at least 4 varchar(4000) fields with each row. Fields that NOBODY wanted, NOBODY needed. But it was easier to write * instead of determine the 12 fields up front. OMG.)
Of course you are going to do full table scans. You ASKED for every column of a LOT of rows. In fact, forcing an index scan, caused the query to run slower, as I showed them.
But reducing the column count of the data coming back, and measuring the data size that came back to the program, they SAW the bottleneck.
I am not all BCNF or 3NF. Denormalization makes sense at times.
The "It Depends" answer is simply this: You should create the DB and the queries you are going to use to get to ALL of the data and reports before you build the system. If the queries you run 100,000 times a day require 15 joins, and the queries that you run once a month are trivial, you may have optimized the wrong end of the equation.
Finally, creating materialized views, or reporting snapshots are QUITE reasonable things to do.
The correct format for data while processing MAY NOT be the correct format for data while reporting.
In an OLTP type system, constantly joining across many tables that are constantly being updated will produce HORRIBLE performance if writes block reads, or UNDO/REDO contention exists.
Thanks for that link. A great article. I saved it!
|
|
|
|
|
My fearless leader was/is a DBA. We were talking about indices and I got some interesting lessons I'd never have picked up on my own. When a compound index makes sense, how you write your select statement can use or waste all the value-added speed of the index.
It's his job to know the nuances of that; it's my job to create the connections and interfaces for the users in whatever language or language mix makes sense. As I noted in posts at other times, we'd hash and thrash trough possible solutions, give the whys and wherefores, and then pick the best solution we can come up with, sans egoism. When possible, with the long view. Golden days.
| Ravings en masse^ |
|---|
| "The difference between genius and stupidity is that genius has its limits." - Albert Einstein | | "If you are searching for perfection in others, then you seek disappointment. If you are seek perfection in yourself, then you will find failure." - Balboos HaGadol Mar 2010 |
|
|
|
|
|
I run into that kind of nonsense quite more often than I'd like. My answer is now "Show me the query plan or no deal". 99.9% of the time an index is missing somewhere.
|
|
|
|
|
I had to deal with this same problem until I was asked to rewrite an application in which all of the joins were done in Visual Basic code, rather than in the database. I think the problem was that my boss didn't understand SQL that well and did not know how to write the queries with joins. So, I used JOINs in views and greatly simplified the VB code, even though I had to explain the SQL many times. I had one problem in that our views were in a local database and the tables were in a remote database, requiring that full tables had to be moved across the wire to be joined in the local database. My boss for a long time insisted that I should join the data in the VB code and that somehow that would be faster. I fixed that by joining sub-queries that used column specifications and WHERE clauses to greatly reduce that amount of data that needed to be transferred across the wire before the joins were made. The difference in speed was huge. Of course, that made the SQL more complex, but my boss has finally accepted it after realizing that the VB code would be much more complex and time consuming both to write and execute.
|
|
|
|
|
It depends on your 'relationship' with the data. If you are creating reports for mostly non-changing data, denormalized tables makes sense to you. If you are a developer charged with maintaining data integrity, flexibility, and performance (in that order) then you would certainly favor normalized structures. At any rate, given the speed of things nowadays, the difference between join and no-join would be practically imperceptible...at least on a small scale.
"Go forth into the source" - Neal Morse
|
|
|
|
|
If the CTO doesn't want anyone to use joins then he made the wrong choice when picking a relational database for data storage. Should have gone ISAM.
|
|
|
|
|
No worries.
A former "CTO" of mine blew up when I suggested changing the font instead of redesigning a report for a new column.
(He didn't grasp (initially) that one could "change" fonts on laser output ... I think he still had 1403 impact printers on the brain. I thought of him as "VP of the mainframe".)
Someone once told YOUR "CTO" that a particular job is running long because of "joins"... so therefore, "all joins are bad".
|
|
|
|
|
I'm sure this plan[^] will work out exactly how Dear Chairwarmer thinks it will.
Did you ever see history portrayed as an old man with a wise brow and pulseless heart, waging all things in the balance of reason?
Is not rather the genius of history like an eternal, imploring maiden, full of fire, with a burning heart and flaming soul, humanly warm and humanly beautiful?
--Zachris Topelius
Training a telescope on one’s own belly button will only reveal lint. You like that? You go right on staring at it. I prefer looking at galaxies.
-- Sarah Hoyt
|
|
|
|
|
I am reminded numerous times each day how lucky I am to be living in a "free" country, particularly one where freedom of religion, press, speech, etc., are the foundational principles.
Marc
|
|
|
|
|
Oh, the irony! Oh, wait... 
#SupportHeForShe
Government can give you nothing but what it takes from somebody else. A government big enough to give you everything you want is big enough to take everything you've got, including your freedom.-Ezra Taft Benson
You must accept 1 of 2 basic premises: Either we are alone in the universe or we are not alone. Either way, the implications are staggering!-Wernher von Braun
|
|
|
|
|
From the article: The main point of the lecture was ‘Keep your mouths shut!’” the source said, speaking on condition of anonymity.
Apparently, the idea has gotten off to a rocky start. 
"the debugger doesn't tell me anything because this code compiles just fine" - random QA comment
"Facebook is where you tell lies to your friends. Twitter is where you tell the truth to strangers." - chriselst
"I don't drink any more... then again, I don't drink any less." - Mike Mullikins uncle
|
|
|
|
|
Sounds like many so-called "liberal" US Universities' speech policies.
#SupportHeForShe
Government can give you nothing but what it takes from somebody else. A government big enough to give you everything you want is big enough to take everything you've got, including your freedom.-Ezra Taft Benson
You must accept 1 of 2 basic premises: Either we are alone in the universe or we are not alone. Either way, the implications are staggering!-Wernher von Braun
|
|
|
|
|
It's a fantastic idea. And what a great haircut too. 
"These people looked deep within my soul and assigned me a number based on the order in which I joined."
- Homer
|
|
|
|
|
My Daughter defines Optimism as: North Korea has a Hotel!
|
|
|
|
|
Does she also define masochism as people voluntarily stay in it?
Did you ever see history portrayed as an old man with a wise brow and pulseless heart, waging all things in the balance of reason?
Is not rather the genius of history like an eternal, imploring maiden, full of fire, with a burning heart and flaming soul, humanly warm and humanly beautiful?
--Zachris Topelius
Training a telescope on one’s own belly button will only reveal lint. You like that? You go right on staring at it. I prefer looking at galaxies.
-- Sarah Hoyt
|
|
|
|
|
Awesome! I would.
Well, I asked if it was called "Hotel California" but she missed the reference.
She laughed when I explained the song...
|
|
|
|
|
This is what you get when you play D&D and can't decide between an elf and a halfling.
modified 20-Oct-19 21:02pm.
|
|
|
|
|
I would like to start playing lineage 2 (just wanting to see new version, have been 4 years off), but the damm computer (new install) is still searching for updates (aprox. 36 hours searching right now) so I could not backup system yet (don't like to backup games)
M.D.V. 
If something has a solution... Why do we have to worry about?. If it has no solution... For what reason do we have to worry about?
Help me to understand what I'm saying, and I'll explain it better to you
Rating helpful answers is nice, but saying thanks can be even nicer.
|
|
|
|
|
Nelek wrote: aprox. 36 hours searching right now
Plug your network cable in. 
There are only 10 types of people in the world, those who understand binary and those who don't.
|
|
|
|
|
|
1.212.075 packets in "received", the only thing working online is "searching for updates"
Last time (father-in-law's laptop) was 20 hours  , 550+ updates in the list (installing back win7 from last backup I did to his system after he got "upgraded")
Now is a new desktop for me, sadly I have no backup yet for this hardware and have to get the list from W7-SP1 until today... I would expect a bit more speed.
M.D.V.
If something has a solution... Why do we have to worry about?. If it has no solution... For what reason do we have to worry about?
Help me to understand what I'm saying, and I'll explain it better to you
Rating helpful answers is nice, but saying thanks can be even nicer.
|
|
|
|
|
Clinton or Trump in the very near future. Whom to choose? Dunno, i'm not an american citizen. There are only a few things I like from Americans... hollywood, sexy guitars, nirvana, STP, metallica and microsoft!
Edit: I forgot to include "Chuck Norris"
modified 8-Sep-16 12:32pm.
|
|
|
|
|
Vincent Maverick Durano wrote: hollywood, sexy guitars, nirvana, STP, metallica and microsoft!
Maybe Steve Ballmer can be a write-in.
"Voters! Voters! Voters!"
Marc
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
