|
Hello, I need to work out what seems to be a pretty advanced query for SQL server and despite a lot of books and all the internet, I can't seem to classify this into an easily searchable-for problem to get a handle on it. Any help classifying this would be appreciated:
I am making an interface for quickly searching a database of text documents. I've already done the indexing part (sucking out the unique words etc), it's the searching part that I need to refine.
I have three tables involved:
The first "documents" is a table that contains many plain text documents and each document has a unique identifier.
The second table "srchdict" one is a dictionary index containing two columns, a list of unique words culled from the documents to be searched and a unique identifier column to uniquely identify each word.
The third table is the key between the "srchdict" and the "documents" and is called "srchkey". It contains two columns: each one is a unique identifier, one column is the id from the dictionary table of each unique word found in a document and the other is a unique identifier that indicates which document that word was found in.
So far so good and no problem populating those tables at all.
Here is the problem, without resorting to doing a whole bunch of multiple queries in code (i.e. I want to do it all at the SQL server), is there a way (stored procedure or single query) to pass in a list of words that are being searched for, and return a list of document ID's of documents that only contain *all* the words in the list.
I've done this before in c++ with an Access database and it involved querying for each word separately, and storing the resulting list of matched document id's in another temporary table until the list of search words was exhausted, then querying the temporary results table to pull out only the duplicate document ID's that appear as many times over as the number of search terms (this ensures that only documents that match *all* search words appear in the results, any that do not match all are discarded).
Any help, either along the lines of "that's the only way to do it" or just a classification of what type of query algorithm etc this is would be tremendously appreceated.
|
|
|
|
|
You could try something like:
Select DocId, Count(Distinct WordKey) From srchkey<br />
Where WordKey In (Select WordKey From srchdict<br />
Where WordText In ("list", "of", "words"))<br />
Group By DocId<br />
Having Count(Distinct WordKey) = 3
If you wanted to use a stored-procedure then you could pass the list of words in using a temporary table.
Have you tried using SQL-Server's own free-text indexing?
Regards
Andy
|
|
|
|
|
Thank you Andy! I'll give that a try and look into the text indexing for SQL.
"Things are more like they are now than they ever were before."
-- Dwight Eisenhower
|
|
|
|
|
Hi Andy, just wanted to let you know that works perfectly and saved me a huge amount of time! Thank you again.
(slight modification, can't use Count on a uniqueidentifier so changed to a *)
- Cheers!
|
|
|
|
|
Select DocId, Count(Distinct WordKey) From srchkey
Where WordKey In (Select WordKey From srchdict
Where WordText In ("list", "of", "words"))
Group By DocId
Having Count(Distinct WordKey) = 3
Using the IN keyword is generally pretty slow. Using IN with a SELECT sub-query is even slower. It's generally a good idea to start out (i.e. prototype) like this, to see if you've got the right basic idea, but you'll want to rewrite the query to use JOINs for production use. As the poster noted, this will require a temporary table containing the search terms.
tblDocuments ( id INT PRIMARY KEY, document TEXT )
tblKeywords ( id INT PRIMARY KEY, keyword NVARCHAR(50) )
tblStatistics ( keyword_id INT, document_id INT, frequency INT )
with PRIMARY KEY as combination of keyword_id and document_id
tblCriteria ( term NVARCHAR(50) ) <--- a temporary table containing the search terms
SELECT D.id, count(D.id)
FROM tblCriteria C
LEFT OUTER JOIN tblKeywords K ON K.keyword = C.term
RIGHT OUTER JOIN tblStatistics S ON S.keyword_id = K.id
LEFT OUTER JOIN tblDocuments D ON D.id = S.document_id
GROUP BY D.id
HAVING count(D.id) = count(C.term)
This is what you should start out with. You probably want to rank the results based on the combined frequency of terms (from the tblStatistics.frequency column).
Make sure to have good indeces on the tblKeywords table, actually on all the tables. If the data remains fairly constant, you should not have a problem with indeces.
|
|
|
|
|
For each XVal, YVal in tblCurrent there exists at least one record in tblPrevious. Each XVal, YVal in tblPrevious has at least one record where EndTime will equal @currstart. All Col18 and Col19 values in tblPrevious are non-NULL decimals. All Col18 and Col19 values in tblCurrent are NULL.
Given these conditions can anyone explain why after running the following stored procedure Col18 will contain values for each XVal, YVal; but Col19 will sometimes contain NULL for a YVal greater than some apparently random number?
PROCEDURE CopyIDs
DECLARE @myXVal INTEGER,
DECLARE @currstart DATETIME
AS
DECLARE @myCntr INTEGER
SET @myCntr = 1
WHILE (@myCntr <= 250)
BEGIN
UPDATE tblCurrent
SET Col18 = prev.Col18, Col19 = prev.IDCol19
FROM tblPrevious prev INNER JOIN tblCurrent curr
ON prev.XVal = curr.XVal AND prev.YVal = curr.YVal
AND prev.EndTime = @currstart
WHERE curr.XVal = @myXVal AND curr.YVal = @myCntr
SET @myCntr = @myCntr + 1
END
About 80% of the time, the values will be copied for both fields. For The other 20%, after the first 60 to 200 records have been updated correctly the remainder of the records will contain NULL in the field Col19.
This procedure is typically called multiple times with different XVals. When there is a failure, 95% of the time only the first call will fail to copy Col19 correctly.
>>>-----> MikeO
|
|
|
|
|
After further review I realize that both Col18 and Col19 fail to copy. Another process overwrites the value in Col18 very shortly after the update fails.
Sorry for my confusion, but the question remains. Why would a stored procedure fail after the first several iterations through the loop?
>>>-----> MikeO
|
|
|
|
|
Here is my situation...
I have to develop tables for a specific product. Currently I have a "Product" table which has the "ModelId", "Features", etc. There is also a table I have called "ProductDetails" which some special performance data.
The problem is, this product can be "Mounted" in three different configurations. The product information for each of the three different mounts are identical but they have a different "ProductId".
What is the best way to design this?
Thanks
Mark Sanders
sanderssolutions.com
|
|
|
|
|
Why not have a Configuration table that specifies the attributes for the configuration. Then put a Configuration ID in the Product or ProductDetails table, which ever one makes more sense to you, to link the configuration to the productId.
Jeremy Oldham
|
|
|
|
|
Hi all
I have a problem with reading arabic data from an Access 2000 DB File in asp 3.0
the problem takes two forms:
1) when I add some Arabic data to the access file and upload it to the server,,I get some rubish when I read the data using ASP 3.0 on the server .. while I can read it when using localhost.
2)When I add data in arabic to the database using an ASP form I can read it again using ASP and works fine .. but when I downlaod the file and open it in Access or asp in localhost I see some rubish again..
Is it a problem of collations , locales etc .. ?
please me help finding a solution..And send your questions if you find my post unclear..
|
|
|
|
|
It goes well when I set Session.Codepage to an appropriate value.
Session.CodePage = 1256
|
|
|
|
|
is there a way to install a msde database using an instance name that doesnt put the computer name in front of it? basically if you dont put an instance name in the setup ini file then the name of the instance will be your computer name. so i gave it one and now instead of the instance being say Developer1 its is now Developer1\Sql1 i have even tried completely uninstalling sql and installing only the msde but to no avail anyone have any ideas?
|
|
|
|
|
With the latest SqlServer (sp3), If no instance name is specified, the default instance name MSSQLSERVER is used.
However, the location will always be MyComputersName\MySqlServerName. This enables you to access the server across a network and is standard format for network connections like mapping drives etc.
Why is this a problem?
Regards
Wayne Phipps
Time is the greatest teacher... unfortunately, it kills all of its students
|
|
|
|
|
Greetings,
I run this code code on a sql 2000 server full version "CREATE DATABASE MYTEST_DB", and it creates my
database fine.
But when I run the code on a desktop version of it I get an error message of "DB_E_ERRORSINCOMMAND"!
Anyone have any ideas as to what I am doing wrong? I was under the impression that what I develop under
a full version should work under the desktop version, was this wrong?
Thanks in advance.
|
|
|
|
|
For some reason the desktop version required me to issue "USE MASTER" first before I did the create whereas the full version didn't do this. After I added that the call worked.
|
|
|
|
|
I am having problems fetching text column from sybase db table.
I have used getbytes, getchars & getstring with oledb.datareader.
I have also used dataset with oledb.dataadpater. Nouse.
Anyone who has used oledb, ado.net to fetch text columns. Is it working for you. If so can you give me any insight on how you got it wotking, what version of oledb are u using.
Thanks!!
Vinay
|
|
|
|
|
I was looking for a SQL query that will allow me to take a 9 digit numeric value in a varchar field and format it in a standard social security format, ie: ###-##-####. So basically I'd like to update all of the migrated records, some of which have the dashes already, some of which don't, and standardize the format. I could of course do a select * from the table, and write a program to do this one row at a time, but I would rather do this globally with a query if possible.
Thanks,
Steven.
|
|
|
|
|
I'm trying to use a Recordset for returning custom data to the client. The data does not come from the database, it is created on the fly from various data.
ADO does support this kind of Recordsets, but all samples I've found are in VB, and the C++ samples just detach the Recordset from the source.
I just create a new Recordset object, add the columns to the Fields collection with Append(...).
I am stuck at the Open() call which requires a connection, but that is exactly what I dont have.
Help? Please?
Off to Brazil in a few days
|
|
|
|
|
Ok, if anyone ever wants to know:
use
Open(vtMissing, vtMissing, adOpenUnspecified, adLockUnspecified, adCmdUnspecified);
Off to Brazil in a few days
|
|
|
|
|
Hello all,
What are the date functions in VBA? I have a string, and I'd like it to hold today's date. What do I call to get the date?
Dim theDate as string
theDate = date()
I know that is wrong, but what do I do?
Thanks
Nick
|
|
|
|
|
I'm not too sure about vba:
In VB it's:
Dim strDate as String
strDate = Format("dd-mmm-yyyy", Now()) ' replace with your own format
|
|
|
|
|
I have a query regarding how to backup and restore single database tables in MS SQL Server through Visual C++ programming code. Is it possible to do so? I have read in MSDN about backing up single database files but i have no clear concept about database files. Are they referring to tables by files?
|
|
|
|
|
I don't recall there beeing any functionality in SQL server to backup a single table without planning for it first. It has to be planned for by creating filegroups and files.
- A filegroup is a collection of one or more files.
- A file is a file in the filesystem.
- On SQL Server you can control wich filegroup(s) a table is stored in.
- SQL Server allows you to control what file(s) or filegroup(s) to backup.
The prosess goes something like this:
1. Create a database filegroup containing one file in your database.
ALTER DATABASE MyDatabase<br />
ADD FILE TO FILEGROUP MyOtherFilegroup<br />
(<br />
NAME = MyOtherFile,<br />
FILENAME = 'c:\Program Files\Microsoft SQL Server\MSSQL\Data\mydb_otherfile.ndf',<br />
SIZE = 5MB,<br />
MAXSIZE = 100MB,<br />
FILEGROWTH = 5MB<br />
)
2. Create (or move) the table on that database file.
CREATE TABLE Tablename(<your table definition here>) ON MyOtherFilegroup
or (to move a table from one filegroup to another) have a quick look in SQL Server Books Online. Select the index and lookup "filegroups, switching filegroup for table".
3. Take a backup of that specific file or filgroup.
The backup command will look something like either
BACKUP DATABASE MyDatabase FILE = 'MyOtherFile' TO 'MyBackupDevice'
or
BACKUP DATABASE MyDatabase FILEGROUP = 'MyOtherFilegroup' TO 'MyBackupDevice'
|
|
|
|
|
You could also use the BCP (Bulk Copy Program) utility along with an script to create or alter the table. (Automatically made for you with Visual Studio .NET).
John
|
|
|
|
|
Hello all,
I have what I hope is a quick question. In my Access 2000 database I have a Table called Candidates. In this table there is a Field called LName. How can I get the data in this field without opening the table?
This what I am trying to do. In the Main Form of my database, you can eneter a Last Name and hit the Command button and it will open the Candidate Table to the first occurence of the person with that last name. I'd like to write an autofill ability in the Main Form. This way when I type A into the space where you type in the last name, it will automatically fill the text to Aardvark for instance. And then if I hit another letter, like b and the string in there is "ab" it would return the first last name with Ab, for instance Aba.
Now, I know the logic behind how to do what I want, but I need the syntax.
How can I make a string equal the text in the last name field?
Would it be:
Dim lastName as String
lastName = [Candidate].[LName] ??
Any help you can offer me would be great.
And one more quick question.
Is there a way to access the characters in a string?
Can I type lastName[1] and get the first letter of the string?
Again thank you very much.
Nick
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
