Commit 854eabea authored by Omran Saleh's avatar Omran Saleh
Browse files

minor changes

parent d4229da4
To provide a higher abstraction level, we have developed a Web application called \textbf{AutoStudio}. It is a user friendly and easy to use browser-based application to run cross-platform using HTML5, Draw2D touch\footnote{\url{http://www.draw2d.org/draw2d/index.html}}, and node.js\footnote{\url{http://www.nodejs.org}}. AutoStudio has several functionalities:
\begin{itemize}
\item It enables users to leverage the emerging \PipeFlow language graphically via a collection of operators (represented by icons) which could be simply ``dragged and dropped'' onto a drawing canvas. The user can assemble the operators in order to create a dataflow graph in a logical way and visually show how they are related, and from this graph, equivalent \PipeFlow script can be generated. By clicking on the operator icon, a pop-up window appears to let the user specify the parameters of \PipeFlow operators, which are required. Moreover, the user can display the help contents for each operator.
\item Contacting the \PipeFlow system to generate the right script (e.g., Storm, Spark, or PipeFabric scripts) based upon the user's selection of language from the dashboard page. This makes the user to be not aware of any of stream-processing languages syntax including \PipeFlow and their constructs. By this application, the user can trigger the execution of the script through the \PipeFlow system via calling the respective engine. Moreover, it handles real-time stats including execution and performance results sent by \PipeFlow system when the script is in execution. When the execution is complete, the application can send an email to the user.
\item Contacting the \PipeFlow system to generate the right script (e.g., Storm, Spark Streaming, or PipeFabric scripts) based upon the user's selection of language from the dashboard page. This makes the user to be not aware of any of stream-processing languages syntax including \PipeFlow and their constructs. By this application, the user can trigger the execution of the script through the \PipeFlow system via calling the respective engine. Moreover, it handles real-time stats including execution and performance results sent by \PipeFlow system when the script is in execution. When the execution is complete, the application can send an email to the user.
\item It provides the options of saving the generated scripts or flow-designs for future reference, loading the saved script and executing it whenever required.
\item An evaluation tool for the generated scripts where the user is interested in comparing as well as evaluating the performance of stream- processing systems in terms of throughput, latency, and resource consumption such as CPU and memory. The evaluation can be performed online using dynamic figures or offline using static figures.
\end{itemize}
......
......@@ -7,6 +7,5 @@ financial data analysis, traffic management, telecommunication monitoring, envir
\item Providing a flexible architecture for users for extending the system by supporting more engines as well as new operators. These extensions should be integrated in the system smoothly.
\item Developing a front-end web application to enable users who have little experience in the \PipeFlow language to express the script or the program and its associated processing algorithm and data pipeline graphically.
\end{itemize}
One of the useful applications for this approach is helping the developers to evaluate various stream processing systems. Instead of writing several scripts, which should perform the same task, for different systems manually, writing a single script in our approach will help to get the same result faster and more efficiently.
One of the useful applications for this approach is helping the developers to evaluate various stream processing systems. Instead of writing several scripts, which should perform the same task, for different systems manually, writing a single script in our approach will help to get the same result faster and more efficiently.
The remainder of the paper is structured as follows: In Sect.~\ref{sec:pipeflow}, we introduce the \PipeFlow language, the system architecture, and an example for the translation between scripts. Next, in Sect.~\ref{sec:app}, we describe our front-end application and give details about its design and provided functionalities. Finally, the planned demonstration is described in Sect.~\ref{sec:demo}.
......@@ -76,7 +76,7 @@ developed for processing massive data streams such as Storm and Spark Streaming.
\category{H.4}{Information Systems Applications}{Miscellaneous}
\category{I.5}{Pattern Recognition}{Miscellaneous}
\keywords{Data stream processing, Autostudio, Query processing, Spark, Storm, Flink, PipeFlow}
\keywords{Data stream processing, Autostudio, Query processing, PipeFabric, Spark Streaming, Storm, Flink, PipeFlow}
\section{Introduction}
\label{sec:intro}
......
......@@ -61,7 +61,7 @@ where \texttt{limit} parameter limits the number of results returned.
\caption{\PipeFlow architecture}
\label{fig:arch}
\end{figure}
In our approach, we have adopted the automated code translation (ACT) technique by taking an input source script written in \PipeFlow language and converting it into an output source script in another language. In general, \PipeFlow system is written in Java and depends heavily on ANTLR\footnote{\url{http://www.antlr.org}} and StringTemplate\footnote{\url{http://www.stringtemplate.org}} libraries. The former libraries generates a \PipeFlow language parser that can build and walk parse trees whereas the latter generates code using pre-defined templates. Basically, the following components are used to achieve the translation of \PipeFlow source script to equivalent target scripts (PipeFabric, Spark, or Storm): \emph{(1)} parser \emph{(2)} flow graph \emph{(3)} template file and \emph{(4)} code generator. The latter two components are specific to target scripts and differ from each other depending on the target code to be generated. And thus for every target language we need to create a separate template file and code generator to generate its code.
In our approach, we have adopted the automated code translation (ACT) technique by taking an input source script written in \PipeFlow language and converting it into an output source script in another language. In general, \PipeFlow system is written in Java and depends heavily on ANTLR\footnote{\url{http://www.antlr.org}} and StringTemplate\footnote{\url{http://www.stringtemplate.org}} libraries. The former libraries generates a \PipeFlow language parser that can build and walk parse trees whereas the latter generates code using pre-defined templates. Basically, the following components are used to achieve the translation of \PipeFlow source script to equivalent target scripts (PipeFabric, Spark Streaming, or Storm): \emph{(1)} parser \emph{(2)} flow graph \emph{(3)} template file and \emph{(4)} code generator. The latter two components are specific to target scripts and differ from each other depending on the target code to be generated. And thus for every target language we need to create a separate template file and code generator to generate its code.
\textbf{Role of Components}: The roles and functionalities of each above-mentioned components are described below and shown in Fig. \ref{fig:arch}.
\begin{description}
......
......@@ -11,8 +11,8 @@ howpublished = {\url{http://infolab.stanford.edu/stream/}},
}
@misc{spark,
author = {Apache Spark},
title = {Spark: Lightning-fast cluster computing},
howpublished = {\url{https://spark.apache.org}},
title = {Spark Streaming},
howpublished = {\url{https://spark.apache.org/streaming/}},
}
@misc{flink,
author = {{Apache Flink}},
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment