Download solution and unpack to "D:\LogDatabases". The solution includes cmd scripts, scimoredb binaries, plugins(import.dll and managed FeedReader.dll with source code), SQL Scripts and asp.NET files.
To install the metadata(database "LogSearch"), open ScimoreDB query window, and execute "solution.zip\SQLScripts\LogSearch.sql" content. Additionally, the script will create a multidimensional index to index facts (windows eventlog, application nlog,...). Database defines stored procedures to add sources and create dimensions.
When adding sources, there are few rules to follow:
The source database must contain 6 tables called Log1 to Log6. As time progresses, facts populated to tables: Log1 -> Log2 ->.. -> Log6 -> Log1. Each Log table will held 30 days of data (configurable), and move to next Log table. When Log6 table moves, Log1 is truncated. You can change the 30 days value in metadata "ActiveBlockDetails" table column "BlockActiveDays". Don't forget to set PARTITION attribute per Log table, in order to distribute the facts in the cluster (see RSSFeed.sql)!
Implement SynchronizeBlock stored procedure. The procedure called by metadata's procedure Synchronize. And it is meant to populate the facts. The parameter @blockId identify the active Log[1-6] table, and, @startLogId the sequence number - the point to index facts by multidimensional index. For example, if the SQL table contains Identity column, using it as sequence number will allow reading X initial rows first time, and then read new rows providing where clause identField > @startLogId. For log file sources, the sequence number will be the size of the log file. When the log file grows, we index only the new rows from the last obtained position.
Add RSS feeds source. The example works out of the box. It is recommended example to start.
The examples of how to index windows event log. The example works out of the box too.
A few examples how to index other sources (SQL Server, Log text,..).
When source(s) (RSS feeds) has been added, perform the initial load (first time synchronization):
The Synchronize procedure will copy and index the changes. Note: it takes 1 minute before new fact appears in search (multidimensional index).
To verify the solution works, execute:
exec logSearch.SearchEntities '+Created:2013*'
For periodic updates, you may create windows scheduled task that executes Synchronize procedure:
"d:\logdatabases\isql.exe". Arguments: --server=localhost --port=999 --database=LogSearch --file="D:\Logdatabases\SynchronizeActivityLog.sql"
"d:\logdatabases\isql.exe". Parameters: --server=localhost --port=999 --database=LogSearch --file="D:\Logdatabases\OptimizeIndexes.sql"