» Scimore News Blog

New release with improvements in analysis of big data

2013-11-25 - Marius Slyzius

Hi Everyone

We have been working on a project to index large amount of unstructured data from event-logs, files, databases etc. This has lead to numerous improvements in the scimore engine, in the area of analysis of big data.

Free text improvements

Scimore have had improvements to its free text search engine, enabling it handling very large indexes. Additionally the following features have been added:

  • Groupby / facets had major improvements to performance. Scimore can now do grouping by multiple fields, and can do group by over 100mio's of rows very fast.

  • Aggregatable measures such as sum/avg/min/max. Ad hoc aggregations for very large data sets.

C# plugins

Scimore can now load and execute managed plugins. Plugins can be used as agents for accessing external data.

New solution/example

Solution/Example a la dataware house indexing multiple external data sources for search and analysis. Example shows how to use the new features, indexing external data sources made available using C# plugins. The sample shows how to handle large unstructured data, and expose search and analysis capabilities similar to SQL Server Analysis Services. The solution demonstrates usage of distributed scimoredb, and handles billions of rows of data. You can find the solution here. You can also see an online version of the sample indexing RSS feeds

As usual you can grab the installer on the download page.

And a Merry Christmas :)