Loghost

Provides centralized logging infrastructure inside a project including remote rsyslog and ELK (Elasticsearch, Logstash, Kibana).

Please refer to chapter Logging for configuration examples and hints on using the ELK stack.

Components

Default setup

  • Elasticsearch, Logstash and Kibana are installed on the loghost.
  • logstash creates elasticsearch indices of the pattern logstash-YYYY.MM.DD.
  • elasticsearch-curator deletes indices older than 90 days.
  • logstash-forwarder ships locally generated logs from all machines to the loghost.
  • rsyslog forwards all syslog entries to the loghost.

Configuration

Below is a list of your configuration entry points for the involved components.

Warning

All configuration needs to be performed as a service user.

  • rsyslog: /etc/rsyslog.d/SNIPPET.conf
  • logstash-forwarder: /etc/logstash-forwarder/conf.d/SNIPPET.conf
  • logstash: /etc/logstash/conf.d/SNIPPET.conf

Interaction

  • rsyslog: sudo /etc/init.d/rsyslog restart for restarts after configuration changes

    Note

    rsyslog ignores invalid configuration statements, so be sure to check /var/log/messages for errors after a restart.

  • logstash-forwarder: sudo /etc/init.d/logstash-forwarder restart for restarts after configuration changes

  • Logstash: sudo /etc/init.d/logstash restart for restarts after configuration changes

  • Kibana: refer to our Logging section for how to interact with Kibana

Monitoring

We monitor for:

  • running processes
  • reachable ports
  • correctly written log files
  • correctly pruned Elasticsearch indices