docker-elk

github源码
Bringing up the stack
Clone this repository onto the Docker host that will run the stack, then start services locally using Docker Compose:
$ docker-compose up

5000: Logstash TCP input
9200: Elasticsearch HTTP user:elastic password:changeme
9300: Elasticsearch TCP transport 集群通信保持心跳
5601: Kibana      user:elastic password:changeme

LogStash 离散的日志收集,处理,过滤,格式转换
ElasticSearch 搜索引擎,进行索引,方便检索
Kibana 展示数据

springboot集成logbacku将日志输出到logstash

<!-- https://mvnrepository.com/artifact/net.logstash.logback/logstash-logback-encoder -->
<dependency>
    <groupId>net.logstash.logback</groupId>
    <artifactId>logstash-logback-encoder</artifactId>
    <version>6.3</version>
</dependency>

grok表达式

logstash.config
filter{
    grok{
        match => {
            "message" => "%{TIMESTAPM_ISO8601:logTime} %{GREEDYDATA:logThread} %{LOGLEVEL:logLevel} %{GREEDYDATA:loggerClass} - %{GREEDYDATA:logContent}"
        }
    }
}

在ELK前使用消息队列提高可用性,可分享性,如Kafka
配制LogStash从kafka接收
配制springboot项目发送到kafka

<!-- https://mvnrepository.com/artifact/com.github.danielwegener/logback-kafka-appender -->
<dependency>
    <groupId>com.github.danielwegener</groupId>
    <artifactId>logback-kafka-appender</artifactId>
    <version>0.2.0-RC2</version>
</dependency>
建立日志配制文件logback-spring.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE configuration>
<configuration>

    <!--将日志发送到LOGSTASH-->
   <!-- <appender name="LOGSTASH" class="net.logstash.logback.appender.LogstashAccessTcpSocketAppender">
        <destination>localhost:5000</destination>-->
        <!--整个日志信息转为一个JSON发送-->
       <!-- <encoder charset="UTF-8" class="net.logstash.logback.encoder.LogstashEncoder"/>-->
        <!--自定义一个发送格式-->
       <!-- <encoder>
            <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{50} - %msg%n</pattern>
            <charset>UTF-8</charset>
        </encoder>
    </appender>-->

    <!--将日志发送到kafka-->
    <appender name="KAFKA" class="com.github.danielwegener.logback.kafka.KafkaAppender">
        <encoder>
            <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{50} - %msg%n</pattern>
        </encoder>
        <topic>test</topic>
        <keyingStrategy class="com.github.danielwegener.logback.kafka.keying.NoKeyKeyingStrategy"/>
        <deliveryStrategy class="com.github.danielwegener.logback.kafka.delivery.AsynchronousDeliveryStrategy"/>
        <producerConfig>bootstrap.servers=192.168.88.108:9092</producerConfig>
        <appender-ref ref="CONSOLE"/>
    </appender>

    <include resource="org/springframework/boot/logging/logback/base.xml"/>

    <root level="INFO">
        <!--<appender-ref ref="LOGSTASH"/>-->
        <appender-ref ref="KAFKA"/>
        <appender-ref ref="CONSOLE"/>
    </root>

</configuration>