Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
project:rpihadoop [2017/11/02 18:21] licho |
project:rpihadoop [2017/12/21 08:58] (current) licho [Zpracovani Dat] |
||
|---|---|---|---|
| Line 11: | Line 11: | ||
| - Data Analysis | - Data Analysis | ||
| {{project:hadoop-data-analysis-arch.png}} | {{project:hadoop-data-analysis-arch.png}} | ||
| + | |||
| + | {{project:kafka-spark.jpg?650}} | ||
| ==== HDFS ==== | ==== HDFS ==== | ||
| {{project:hadoop-hdfs-arch.png?550}} | {{project:hadoop-hdfs-arch.png?550}} | ||
| Line 679: | Line 681: | ||
| ==== Krok 6: Flume ===== | ==== Krok 6: Flume ===== | ||
| == Prerequisite: == | == Prerequisite: == | ||
| - | * **JDK 1.6 or later versions of Java** installed on our Ubuntu machine. | + | * **JDK 1.6 or later versions of Java** installed on our machine. |
| * **Memory** – Sufficient memory for configurations used by sources, channels or sinks. | * **Memory** – Sufficient memory for configurations used by sources, channels or sinks. | ||
| * **Disk Space** – Sufficient disk space for configurations used by channels or sinks. | * **Disk Space** – Sufficient disk space for configurations used by channels or sinks. | ||
| Line 692: | Line 694: | ||
| - **Edit:** In ''FLUME_CONF_DIR'' directory, rename flume-env.sh.template file to ''flume-env.sh'' and provide value for ''JAVA_HOME'' environment variable with Java installation directory. | - **Edit:** In ''FLUME_CONF_DIR'' directory, rename flume-env.sh.template file to ''flume-env.sh'' and provide value for ''JAVA_HOME'' environment variable with Java installation directory. | ||
| - If we are going to use **memory channels** while setting flume agents, it is preferable to increase the memory limits in ''JAVA_OPTS'' variable. By default, the minimum and maximum memory values are 100 MB and 200 MB respectively (Xms100m -Xmx200m). Better to increase these limits to **500 MB** and **1000 MB** respectively. Shell: <code>JAVA_HOME="cesta" | - If we are going to use **memory channels** while setting flume agents, it is preferable to increase the memory limits in ''JAVA_OPTS'' variable. By default, the minimum and maximum memory values are 100 MB and 200 MB respectively (Xms100m -Xmx200m). Better to increase these limits to **500 MB** and **1000 MB** respectively. Shell: <code>JAVA_HOME="cesta" | ||
| - | JAVAOPTS="-Xms200m -Xmx800m -Dcom.sun/management.jmxremote"</code> | + | JAVAOPTS="-Xms500m -Xmx1000m -Dcom.sun/management.jmxremote"</code> |
| - **Work done:** With these settings, we can consider flume installation as completed. | - **Work done:** With these settings, we can consider flume installation as completed. | ||
| - **Verification:** We can verify the flume installation with<code>$ flume-ng –help</code> command on terminal. If we get output similar to below then flume installation is successful. | - **Verification:** We can verify the flume installation with<code>$ flume-ng –help</code> command on terminal. If we get output similar to below then flume installation is successful. | ||