|
1. The two are different, and you have to go back a few years to when QA was introduced to understand why it was introduced.
2. I did not see anything related to C# in the original question.
Michael Balloni wrote: My first time posting on this forum.
So why say that when he has been, as you pointed out, a contributor of many years standing? And that was why I suggested he was in the wrong place.
|
|
|
|
|
Richard MacCutchan wrote: go back a few years to when QA was introduced to understand why it was introduced.
If you have time, i'd appreciate knowing your perceptions/memories of "why."Richard MacCutchan wrote: 2. I did not see anything related to C# in the original question.
i interpreted this more narrowly: as meaning he had not posted in this forum before. But, i agree that his database article should have been mentioned, and that might have made the post less "ambiguous."
cheers, Bill
«The mind is not a vessel to be filled but a fire to be kindled» Plutarch
|
|
|
|
|
BillWoodruff wrote: perceptions/memories of "why.
IIRC the idea was to attract 'quick' questions. That is to say ones that could be answered fairly easily without long discussion threads as tend to exist in the forums.
|
|
|
|
|
Hi, i'm not trying to go "legal" on you, and i am getting all i ever hoped for, and more  , from CP; but, in your opinion, is it optimum to have (what i perceive as) the kind of overlap we have now ?
If i am the only one beating the gong, i'll just adjust my attitude and perceptions.
cheers, Bill
«The mind is not a vessel to be filled but a fire to be kindled» Plutarch
|
|
|
|
|
Well it is what Chris and the team wanted, and in some ways it works well for both questioner and responder. But the problem with an open forum is getting all the users (myself included) to follow the rules correctly.
|
|
|
|
|
... understood ... thanks !
i've been told i have a real gift for thinking up things other people should change 
«The mind is not a vessel to be filled but a fire to be kindled» Plutarch
|
|
|
|
|
BillWoodruff wrote: a real gift for thinking up things other people should change
I think we can all do that.
|
|
|
|
|
Hi Michael, i'm surprised you didn't mention your 2021 CP article: [^] to which i and other people responded.
imho, your GitHub site has no Wiki; the intro (.md file) is a single statement that does not effectively "advertise" what the project is about, and motivate the viewer to explore it. i suggest adding much more information for the visitor. And, aren't there ways to promote a GitHub site ?
i have no quibble with your posting this here; over the years, this forum has mutated from being a place for C# language issue discussion to being a QA forum, and tolerating whatever else walks through the door.
There is a (relatively inactive) DB forum here: [^] ... why not post there, also ?
imho, code-review and concept-review, to any real depth, are rara avis in this section of the zoo (code-monkey-house ?) That's not a complaint ! Perhaps StackExchange's Code Review forum [^] might be fertile ground for you ?
Wish I had the database experience and depth (and youth !) to evaluate your implementation, and ask intelligent questions.
cheers, Bill
«The mind is not a vessel to be filled but a fire to be kindled» Plutarch
modified 17-Oct-21 5:44am.
|
|
|
|
|
Hey Bill,
Thanks for taking the time. Sorry this is so late, I don't know why CP wasn't emailing me with y'all's responses. Just got a comment here about the general premise, tried to speak to that. Yeah, I haven't fleshed out the GitHub "sites" at all yet, was hoping to get some direction based on my question as to how this tool might best be used, and a code review is always much appreciated. I posted to the Database group, no response there. I'm porting it to C++ for fun, that's been a blast, should make for another fun article.
Cheers, -Michael
|
|
|
|
|
Hi Michael,Michael Sydney Balloni wrote: was hoping to get some direction based on my question as to how this tool might best be used
i think that without more of an overview of the project, and summary details of its architecture, api's, and work-flow, its features, its limitations, etc., you may not get the responses that would be useful.
imho, JSchell's reaction, posted here, probably reflects the naturally skeptic attitude people with significant db experience will take tom such innovation.
People like me, without pro-level db experience, may compare what you are doing with Mehdi Gholam's RaptorDB here on CP: [^] a major project developed over years, with extensive performance testing/timing.
If your project has a limited range of functionality ... and, i understand what that is ... and, performance in its range of use-cases is faster, more efficient, easier to use ... that's fine with me !
«The mind is not a vessel to be filled but a fire to be kindled» Plutarch
|
|
|
|
|
|
Michael Sydney Balloni wrote: general premise
"A simple, easy to use, highly productive,"
My feedback is that I doubt that statement.
Mainstream databases have massive amounts of resources available to them. Productivity for the persisted data is often limited by the complexity of the requirements and how to fit that complexity into the generalities of the common persisted data solutions. However that complexity is what one allows one to craft solutions in the first place without doing a deep dive into how persisted data might solve be used to solve that.
Every 'new' persisted data store solution that I have seen introduced in the past 10 years or so has been done so to support a very specific type of data driven need and none of them are actually anything that could not at least be implemented by traditional mainstream persisted solutions (naturally one might suppose there are speed/cost differences.)
Certainly nothing in the above suggests what is the exact need that you think your solution will address? Does it attempt to replace enterprise Oracle installs? Or is it just another way to manage configuration information for a single app? Maybe it is supposed to compete with Elasticsearch because it is faster? Or something completely new that has only been identified recently by changing user driven needs?
|
|
|
|
|
Thanks for having a look, I really appreciate it.
Taking apart the premise...
- File-based is simpler than server-based.
- Four operations are easier to use than dozens.
- And something simpler and easier to use is more productive.
4db.net builds on SQLite, and provides the database functionality needed for basic applications. It's not fast, it's not sophisticated or robust, but if you only need those four statements, 4db.net is the way to go. To see what I'm talking about, check out this small cars-based sample:
4db.net/Program.cs at main · michaelsballoni/4db.net · GitHub
Imagine how much SQLite code that would take, let alone the server setup for MySQL or some other RDBMS.
Sometimes you need a yacht. But sometimes you just need a canoe. What do you think?
|
|
|
|
|
Michael Sydney Balloni wrote: Sometimes you need a yacht. But sometimes you just need a canoe.
and, some people may need to understand what a paddle is, and why a canoe is a good design for a boat
«The mind is not a vessel to be filled but a fire to be kindled» Plutarch
|
|
|
|
|
Michael Sydney Balloni wrote: Imagine how much SQLite code that would take, let alone the server setup for MySQL or some other RDBMS.
Not sure you really understood my point though.
I work on large applications. So the amount of code and even management of the persisted store data systems is not significant because the applications are going to be large regardless.
And even midsize solutions usually have aspirations to be bigger and will likely need more functionality than provided.
So based on the requirements presented here this only works for very small systems. And for those systems any solution is probably going to be adequate.
So as I suggested before without a specifically identified niche I don't see the need for what you are suggesting.
|
|
|
|
|
Point taken. In .NET, coding directly against SQLite is easy and you get great functionality. No brainer there.
So I ported the .NET metastrings / 4db stuff to C++, where coding against the C SQLite library is a pain.
I wrote this:
4db: A Dynamic File-based NoSQL Database for C++
For a C++ programmer wanting basic record persistence and not much else, this is a good alternative to fprintf or hacking SQLite by hand. In that article, in its code, there are a couple wrapper classes for SQLite. Those might have more valuable than 4db. It's been a wild ride...
|
|
|
|
|
I have some folders having files, its count like 40Lac, 70Lac, 1Cr, 1.5cr. And filetype like .png, .xlsx, .txt, .msg, .ico, .jpg, .bmp etc...
Now i want to insert this filename into my database table with its size, but when i implement it take too much time to scanning the folder, and then after it throws memory exception.
can anyone please help me out here, how can i implement this scenario in a better manner.
how can insert faster in table ?
I am using C#.net with PostgreSQL database.
Ankur B. Patel
|
|
|
|
|
First of all, what is a lac and cr?
This is never going to be "fast", more important is that you do it correct. I assume your putting all your sh*t in a blob. Don't, txt should be archived as memo, so you can later use search form DB.
If you just want to archive names and their sizes, read the entire folders' contents and spawn some threads to save chuncks of that.
Bastard Programmer from Hell 
"If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.
|
|
|
|
|
|
Yah. So them like Americans with their pounds.
In international communication we use the SI system. If you can't, better learn
Bastard Programmer from Hell
"If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.
|
|
|
|
|
Thanks but, I just correct here that forget about lac and cr (it's Lakh and Crore), keep in mind that you have millions of files in a folder and I just want to insert the files name and its size in bytes into the database.
how do I bulk insert? and also keep in mind that system memory doesn't go high.
please suggest to me, how I can do it quickly and securely.
|
|
|
|
|
Pipe a DIR listing to a text file. That'll give you the list you need; which you can then "substring" and load into a table of files names, etc.
Displays a list of files and subdirectories in a directory.
DIR [drive:][path][filename] [/A[[:]attributes]] [/B] [/C] [/D] [/L] [/N]
[/O[[:]sortorder]] [/P] [/Q] [/R] [/S] [/T[[:]timefield]] [/W] [/X] [/4]
[drive:][path][filename]
Specifies drive, directory, and/or files to list.
/A Displays files with specified attributes.
attributes D Directories R Read-only files
H Hidden files A Files ready for archiving
S System files I Not content indexed files
L Reparse Points O Offline files
- Prefix meaning not
/B Uses bare format (no heading information or summary).
/C Display the thousand separator in file sizes. This is the
default. Use /-C to disable display of separator.
/D Same as wide but files are list sorted by column.
/L Uses lowercase.
/N New long list format where filenames are on the far right.
/O List by files in sorted order.
sortorder N By name (alphabetic) S By size (smallest first)
E By extension (alphabetic) D By date/time (oldest first)
G Group directories first - Prefix to reverse order
/P Pauses after each screenful of information.
/Q Display the owner of the file.
/R Display alternate data streams of the file.
/S Displays files in specified directory and all subdirectories.
It was only in wine that he laid down no limit for himself, but he did not allow himself to be confused by it.
― Confucian Analects: Rules of Confucius about his food
|
|
|
|
|
If this is your idea of a backup mechanism, don't bother: it'll never be quick, it'll never be efficient, and it will always risk running out of memory.
Instead, think about using a "proper" backup system which archives the disk as sectors instead of files - it's a lot quicker, a lot more efficient, and much, much safer.
Remember, backups should be air-gapped for safety: DB's and suchlike are just files and as such are just as much at risk from ransomware and any other data (more so, in some cases as they are a "Prime target" for ransomware to exploit).
I use AOMEI Backupper - it has a free Standard version and it allows you to mount the backup images as a disk and retrieve individual files if necessary.
"I have no idea what I did, but I'm taking full credit for it." - ThisOldTony
"Common sense is so rare these days, it should be classified as a super power" - Random T-shirt
AntiTwitter: @DalekDave is now a follower!
|
|
|
|
|
Hi Ankur,
I'm going to assume your files are in a bunch of directories, not all in one, or mostly in a few, but leafy. I'd use the System.IO classes Directory and Path and FileInfo. You can use Directory.GetDirectories to get then directories, then loop over those directory paths calling Directory.GetFiles to get file paths to the files, then for each file path call Path.GetFilename to get the filename, and (new FileInfo(filePath)).Length to get the file length. I'd use a DB transaction per directory, and create an INSERT statement per file and execute it inside the transaction, and commit when you're done with the files in that directory.
Things not to do:
Don't get all files in all directories.
Don't build one SQL statement for all the files.
Hope this helps, -Michael
|
|
|
|
|
Ankur B. Patel wrote: can anyone please help me out here, how can i implement this scenario in a better manner.
Then easiest and best answer do not do that in the first place.
File systems, not databases, have existed for a long time and exist solely to manage files. What you have are files.
If you want to reference the files in the database then do just that. Keep the files on the file system and provide a reference (absolute or relative) to the location of the file.
You can keep meta data on the file in the database, such as name, size, type, etc.
Might also want to insure uniqueness also. You do that by finding out what makes one file always different from other. Usually some combination of name and location. And insure that is maintained.
Finally of course how will these files be used. For example if you expect a png to show up in a web page that has 10,000 unique visits a day, you certainly do not want to be pulling it out of a database.
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
