|
Reaching out to the community to pressure test our internal thinking.
We are building a simplified business intelligence platform that will aggregate metrics (i.e. traffic, backlinks) and text list (i.e search keywords, used technologies) from several data providers.
The data will be somewhat loosely structured and may change over time with vendors potentially changing their response formats.
Data volume may be long term 100,000 rows x 25 input vectors.
Data would be updated and read continuously but not at massive concurrent volume.
We'd expect to need to do some ETL transformations on the gathered data from partners along the way to the UI (e.g show trending information over the past five captured data points).
We'd want to archive every single data snapshot (i.e. version it) vs just storing the most current data point.
The persistence technology should be readily available through AWS.
Our assumption is our requirements lend themselves best towards DynamoDB (vs Amazon Neptune or Redshift or Aurora).
Is that fair to assume? Are there any other questions / information I can provide to elicit input from this community?
|
|
|
|
|
Member 14070096 wrote: Is that fair to assume
No, it is an assumption. Fair would be to evaluate them on their merits, and award points for each merit. My guess is that any NoSQL database would do.
Member 14070096 wrote:
The data will be somewhat loosely structured and may change over time with vendors potentially changing their response formats.
That's wrong; your format should depend on the data that you want to collect, not on the format of various datasources.
Bastard Programmer from Hell 
If you can't read my code, try converting it here[^]
"If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.
|
|
|
|
|
Why a NoSQL database, I would have thought that a relational DB would serve the purpose better.
Never underestimate the power of human stupidity -
RAH
I'm old. I know stuff - JSOP
|
|
|
|
|
Mycroft Holmes wrote: Why a NoSQL database
Good question; his example of Dynamo is, but..
Mycroft Holmes wrote: I would have thought that a relational DB would serve the purpose better.
..is probably true 
Bastard Programmer from Hell
If you can't read my code, try converting it here[^]
"If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.
|
|
|
|
|
You will HAVE to have an ETL layer between your various sources and your database (assuming it is a relational DB). You need to get all your sources into a single format and deal with changing source structures which will need recoding the ETL to suit.
Never underestimate the power of human stupidity -
RAH
I'm old. I know stuff - JSOP
|
|
|
|
|
Hi
I wish to only begin a select statement if the top value date is found - this the first of the month. It's my way of knowing data has been added into a system at the beginning of the month e.g max date is 01/11/2018. If not take the last day of the previous month 31/10/2018. Table is called sales. Column is called SaleDateFrom.
I want to pull e.g data from SaleDateFrom <=01/11/2018...if not pull data
SaleDateFrom <=31/10/2018
statement to change
where (SaleDateFrom<= '20181101'
and (SaleDateTo '20010101' or
SaleDateTo is null))
Any ideas team?
|
|
|
|
|
|
hi
I don't know whats happened sorry...I tried so hard to edit the previous post, and it would not let me, so I signed out , went to log in it said denied, asked for reset I think it has issues if you login with facebook as uses same email. please help
|
|
|
|
|
If you have problems with your account then you should use the Bugs and Suggestions[^] forum to get assistance from the administrators. I suggest you do so now from your original account, and delete this one.
|
|
|
|
|
Not only is this a repost of the question below, you haven't responded to any of the questions asked in response to that post.
2018-10-31 is less than 2018-11-01; therefore, if there are no records with SaleDateFrom <= '20181101', there will be no records with SaleDateFrom <= '20181031'.
You need to explain what you're actually trying to achieve.
"These people looked deep within my soul and assigned me a number based on the order in which I joined."
- Homer
|
|
|
|
|
I'm doing my best to explain ,
here is my where clause
where (SaleDateFrom<= '20181101'
and (SaleDateTo '20010101' or
SaleDateTo is null))
I want to replace the date in bold to 20181031 if the highest date entered into the data base is not the 20181101. this is to make sure the data for the start of the month has been entered
please help
please see response to Richard - about account big issue, which I be contacting code project about
|
|
|
|
|
And once again: If there are no records where the SaleDateFrom is on or before 1st November, then there will be no records where SaleDateFrom is on or before 31st October.
If there is a record for 31st October, then that will already be included in your current query.
If there are no records on or before 1st November, then changing your query to look for records with an earlier date isn't going to make any difference.
"These people looked deep within my soul and assigned me a number based on the order in which I joined."
- Homer
|
|
|
|
|
that makes sense now...fab , I totally get it. thank you team.. I do appreciate the help 
|
|
|
|
|
Why don't you use a sub-select for the date?
where (SaleDateFrom <= ( select max(your date column) from your table with date value ) and (SaleDateTo '20010101' or SaleDateTo is null ))
|
|
|
|
|
Hi
I wish to only begin a select statement if the top value date is found. It's my way of knowing data has been added into a system at the beginning of the month e.g max date is 01/01/2018. If not take the last day of the month 31/10/2018. Table is called sales. Column is called SaleDateFrom.
I want to pull e.g data from SaleDateFrom <=01/01/2018...if not pull data
SaleDateFrom <=31/12/2018
Any ideas team?
|
|
|
|
|
That does not make a lot of sense. If you ask for records whose date is less than or equal to 01/01/2018, then all records whose date is less than 31/12/2017 (I presume you mean 2017 here) will also fulfil the criteria.
|
|
|
|
|
You want to pull the "most recent months" data.
Yes that it possible. Didn't look but I am certain that google will display many answers for that. Add your specific database in do get a more relevant answer.
|
|
|
|
|
I have 2 date time entries that represent time in and time out. I want to graph this data representing how many people or on the clock per 15 minute intervals.
Should I create a temp table that holds the qty of people that are clocked in for every 15 minute interval or what?
|
|
|
|
|
I usually use a view to service this type of requirement. Craft the view to supply only the required data eg date, starttime in 15 minute groupings and the number of people falling into the slots who are clocked on. End time only represents a person not clocked on and is irrelevant to the graph.
Never underestimate the power of human stupidity -
RAH
I'm old. I know stuff - JSOP
|
|
|
|
|
I am trying to document a system that I have taken over
it is a single website with 3 database connections. The access to the database is via a single SQL Server login and it has the minimal access that is required for the site.
My question is; each database has the same logon should they have different logon names for the database?
Thanks
Simon
Every day, thousands of innocent plants are killed by vegetarians.
Help end the violence EAT BACON
modified 9-Nov-18 4:33am.
|
|
|
|
|
My door key fits in the front door and in the back door; it may not be as safe as two keys, but it is convenient. OTOH, there's little of value here, so the risc of something happening is low.
Bastard Programmer from Hell 
If you can't read my code, try converting it here[^]
"If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.
|
|
|
|
|
Why would two keys be safer?
You only need to break into one door to access the house.
|
|
|
|
|
That's because my example is not perfect; if it were two houses it would be more in line with the multiple databases.
I do not buy locks based on best practices, but rather on the risc versus the damage.
Bastard Programmer from Hell
If you can't read my code, try converting it here[^]
"If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.
|
|
|
|
|
|
Thank Eddy your analogy made sense 
Every day, thousands of innocent plants are killed by vegetarians.
Help end the violence EAT BACON
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
