We have been working on a project to index large amount of unstructured data from event-logs, files, databases etc. This has lead to numerous improvements in the scimore engine, in the area of analysis of big data.
Scimore have had improvements to its free text search engine, enabling it handling very large indexes. Additionally the following features have been added:
Groupby / facets had major improvements to performance. Scimore can now do grouping by multiple fields, and can do group by over 100mio's of rows very fast.
Aggregatable measures such as sum/avg/min/max. Ad hoc aggregations for very large data sets.
Scimore can now load and execute managed plugins. Plugins can be used as agents for accessing external data.
Solution/Example a la dataware house indexing multiple external data sources for search and analysis. Example shows how to use the new features, indexing external data sources made available using C# plugins. The sample shows how to handle large unstructured data, and expose search and analysis capabilities similar to SQL Server Analysis Services. The solution demonstrates usage of distributed scimoredb, and handles billions of rows of data. You can find the solution here. You can also see an online version of the sample indexing RSS feeds
As usual you can grab the installer on the download page.
And a Merry Christmas :)