Structured Big Data Analysis on openSUSE (12.3)

This solution is often described as an all-opensource alternative to Splunk, but can be taken in many directions.

The guides I posted describe how to install on openSUSE (12.3 but is not really distro version specific)
**Logstash **- logfile aggregator for any/all types of logfiles generated by practically any app or system. Has embedded Grok (RegEx) that automatically parses many, many common logfile formats and converts to a common format suitable for analysis
Elasticsearch - Mapping and Reduce, it builds the indexes and metadata used to enable fast query searches
Kibana - A poplular and commonly preferred web interface tool that enables easy User queries and displays results in graphs which support clicking on to drill down.

Of more general interest, the Kibana install guide includes a prerequisites list basic to installing ruby applications (gems specifically). It’s interesting a ruby pattern does not exist that ensures all dependencies are installed, but here it is (or should be).

The guide will likely enable any person to be up and running in a virtual or bare metal on a single node within an hour or so. I have also modified the steps for running the Logstash tests to my liking (download all demo/test files at once).

From this install, a large enterprise production deployment can be configured “as is” but can also be scaled horizontally, each part and data can be distributed across any number of nodes. Also, this solution is primarily intended to analyze log data. It’s also possible to input other data types into Elasticsearch.

For any Big Data student or professional, I hope these Guides are usefull… Of course, documenting something this large might be subject to errors, I’d be grateful for anyone to submit any suggestions or errata to me (or just edit the pages, I have no problem with that).

My own openSUSE wiki TOC
https://en.opensuse.org/User:Tsu2
The main Logstash/Elasticsearch/Kibana page
https://en.opensuse.org/User:Tsu2/Install_and_Intro_Logstash-Elasticsearch-Kibana
The individual Install pages
Logstash Install
Elasticsearch Install
Kibana Install

HTH,
'TSU