Apache Flume: distributed log collection for Hadoop design and implement a series of Flume agents to send streamed data into Hadoop

If you are a Hadoop programmer who wants to learn about Flume to be able to move datasets into Hadoop in a timely and replicable manner, then this book is ideal for you. No prior knowledge about Apache Flume is necessary, but a basic knowledge of Hadoop and the Hadoop File System (HDFS) is assumed

Bibliographic Details
Main Author: Hoffman, Steve
Format: eBook
Language:English
Published: Birmingham, UK Packt Publishing 2015
Edition:Second edition
Series:Community experience distilled
Subjects:
Online Access:
Collection: O'Reilly - Collection details see MPG.ReNa
LEADER 05157nmm a2200589 u 4500
001 EB001908894
003 EBX01000000000000001071796
005 00000000000000.0
007 cr|||||||||||||||||||||
008 210123 ||| eng
020 |a 9781784399146 
050 4 |a QA76.9.D5 
100 1 |a Hoffman, Steve 
245 0 0 |a Apache Flume: distributed log collection for Hadoop  |b design and implement a series of Flume agents to send streamed data into Hadoop  |c Steve Hoffman 
246 3 1 |a Design and implement a series of Flume agents to send streamed data into Hadoop 
250 |a Second edition 
260 |a Birmingham, UK  |b Packt Publishing  |c 2015 
300 |a 1 volume  |b illustrations 
505 0 |a Chapter 5: Sources and Channel SelectorsThe problem with using tail; The Exec source; Spooling Directory Source; Syslog sources; The syslog UDP source; The syslog TCP source; The multiport syslog TCP source; JMS source; Channel selectors; Replicating; Multiplexing; Summary; Chapter 6: Interceptors, ETL, and Routing; Interceptors; Timestamp; Host; Static; Regular expression filtering; Regular expression extractor; Morphline interceptor; Custom interceptors; The plugins directory; Tiering flows; The Avro source/sink; Compressing Avro; SSL Avro flows; The Thrift source/sink 
505 0 |a Cover; Copyright; Credits; About the Author; About the Reviewers; www.PacktPub.com; Table of Contents; Preface; Chapter 1: Overview and Architecture; Flume 0.9; Flume 1.X (Flume-NG); The problem with HDFS and streaming data/logs; Sources, channels, and sinks; Flume events; Interceptors, channel selectors, and sink processors; Tiered data collection (multiple flows and/or agents); The Kite SDK; Summary; Chapter 2: A Quick Start Guide to Flume; Downloading Flume; Flume in Hadoop distributions; An overview of the Flume configuration file; Starting up with ""Hello, World!""; Summary 
505 0 |a MonitNagios; Monitoring performance metrics; Ganglia; Internal HTTP server; Custom monitoring hooks; Summary; Chapter 9: There Is No Spoon -- the Realities of Real-time Distributed Data Collection; Transport time versus log time; Time zones are evil; Capacity planning; Considerations for multiple data centers; Compliance and data expiry; Summary; Index 
505 0 |a Using command-line AvroThe Log4J appender; The Log4J load-balancing appender; The embedded agent; Configuration and startup; Sending data; Shutdown; Routing; Summary; Chapter 7: Putting it All Together; Web logs to searchable UI; Setting up the web server; Configuring log rotation to the spool directory; Setting up the target -- Elasticsearch; Setting up Flume on collector/relay; Setting up Flume on the client; Creating more search fields with an interceptor; Setting up a better user interface -- Kibana; Archiving to HDFS; Summary; Chapter 8: Monitoring Flume; Monitoring the agent process 
505 0 |a Chapter 3: ChannelsThe memory channel; The file channel; Spillable Memory Channel; Summary; Chapter 4: Sinks and Sink Processors; HDFS sink; Path and filename; File rotation; Compression codecs; Event Serializers; Text output; Text with headers; Apache Avro; User-provided Avro schema; File type; SequenceFile; DataStream; CompressedStream; Timeouts and workers; Sink groups; Load balancing; Failover; MorphlineSolrSink; Morphline configuration files; Typical SolrSink configuration; Sink configuration; ElasticSearchSink; LogStash Serializer; Dynamic Serializer; Summary 
653 |a Traitement réparti 
653 |a COMPUTERS / Computer Science / bisacsh 
653 |a COMPUTERS / Hardware / General / bisacsh 
653 |a Electronic data processing / Distributed processing / fast 
653 |a COMPUTERS / Data Processing / bisacsh 
653 |a COMPUTERS / Reference / bisacsh 
653 |a Electronic data processing / Distributed processing / http://id.loc.gov/authorities/subjects/sh85042293 
653 |a File organization (Computer science) / http://id.loc.gov/authorities/subjects/sh85048195 
653 |a File organization (Computer science) / fast 
653 |a COMPUTERS / Computer Literacy / bisacsh 
653 |a Fichiers (Informatique) / Organisation 
653 |a Apache Hadoop / fast 
653 |a COMPUTERS / Machine Theory / bisacsh 
653 |a Apache Hadoop / http://id.loc.gov/authorities/names/n2013024279 
653 |a COMPUTERS / Information Technology / bisacsh 
041 0 7 |a eng  |2 ISO 639-2 
989 |b OREILLY  |a O'Reilly 
490 0 |a Community experience distilled 
500 |a Includes index 
776 |z 9781784399146 
776 |z 1784399140 
776 |z 1784392170 
776 |z 9781784392178 
776 |z 1784399140 
856 4 0 |u https://learning.oreilly.com/library/view/~/9781784392178/?ar  |x Verlag  |3 Volltext 
082 0 |a 500 
082 0 |a 004.36 
082 0 |a 300 
520 |a If you are a Hadoop programmer who wants to learn about Flume to be able to move datasets into Hadoop in a timely and replicable manner, then this book is ideal for you. No prior knowledge about Apache Flume is necessary, but a basic knowledge of Hadoop and the Hadoop File System (HDFS) is assumed