Commit 8b3a3fd5 authored by Omran Saleh's avatar Omran Saleh
Browse files

the submitted version

parent f2f0dbc5
To provide a higher abstraction level, we have developed a Web application called \textbf{AutoStudio} as shown in Fig. \ref{fig:auto}. It is a user friendly and easy to use browser-based application to run cross-platform using HTML5, Draw2D touch\footnote{\url{http://www.draw2d.org/draw2d/index.html}}, and node.js\footnote{\url{http://www.nodejs.org}}. AutoStudio has several functionalities: To provide a higher abstraction level, we have developed a Web application called \textbf{AutoStudio} as shown in Fig. \ref{fig:auto}. It is a user friendly and easy to use browser-based application to run cross-platform using HTML5, Draw2D touch\footnote{\url{http://www.draw2d.org/draw2d/index.html}}, and node.js\footnote{\url{http://www.nodejs.org}}. AutoStudio has several functionalities:
\begin{itemize} \begin{itemize}
\item It enables users to leverage the emerging \PipeFlow language graphically via a collection of operators (represented by icons) which could be simply ``dragged and dropped'' onto a drawing canvas. The user can assemble the operators in order to create a dataflow graph in a logical way and visually show how they are related, and from this graph, equivalent \PipeFlow script can be generated. By clicking on the operator icon, a pop-up window appears to let the user specify the parameters of \PipeFlow operators, which are required. Moreover, the user can display the help contents for each operator. \item It enables users to leverage the emerging \PipeFlow language graphically via a collection of operators (represented by icons) which could be simply ``dragged and dropped'' onto a drawing canvas. The user can assemble the operators in order to create a dataflow graph in a logical way and visually show how they are related, and from this graph, equivalent \PipeFlow script can be generated. By clicking on the operator icon, a pop-up window appears to let the user specify the parameters of \PipeFlow operators, which are required. Moreover, the user can display the help contents for each operator.
\item Contacting the \PipeFlow system to generate the right program (e.g., Storm, Spark Streaming, or PipeFabric programs) based upon the user's selection of engine from the dashboard page. This makes the user to be not aware of any of stream-processing language syntax and their constructs including \PipeFlow. By this application, the user can trigger the execution of the script through the \PipeFlow system via calling the respective engine. Moreover, it handles real-time stats including execution and performance results sent by the \PipeFlow system when the program is in execution. When the execution is complete, the application can send an email to the user. \item Contacting the \PipeFlow system to generate the right program (e.g., Storm, Spark Streaming, or PipeFabric programs) based upon the user's selection of engine from the dashboard page. This makes the user to be not aware of any of stream-processing language syntax and its constructs including \PipeFlow. By this application, the user can trigger the execution of the script through the \PipeFlow system via calling the respective engine. Moreover, it handles real-time stats including execution and performance results sent by the \PipeFlow system when the program is in execution. When the execution is complete, the application can send an email to the user.
\item It provides the options of saving the generated scripts or flow-designs for future reference, loading the saved script and executing it whenever required. \item It provides the options of saving the generated scripts or flow-designs for future reference as well as loading the saved script and executing it whenever required.
\item An evaluation tool for the generated scripts where the user is interested in comparing as well as evaluating the performance of stream- processing systems in terms of throughput, latency, and resource consumption such as CPU and memory. The evaluation can be performed online using dynamic figures or offline using static figures. \item An evaluation tool for the generated programs where the user is interested in comparing as well as evaluating the performance of stream- processing systems in terms of throughput, latency, and resource consumption such as CPU and memory. The evaluation can be performed online using dynamic figures or offline using static figures.
\end{itemize} \end{itemize}
AutoStudio prominently uses open source softwares and frameworks. The client side, including HTML5, JavaScript, Cascading Style Sheets (CSS), jQuery (and helping libraries), twitter bootstrap, and hogan.js, is used for building the graphical user interface, performing Ajax requests, file uploading and downloading, etc. AutoStudio extensively uses pre-compiled hogan templates where the data returned from the server is simply passed to these templates for quick rendering. In addition, Draw2D touch is used to enable creation of diagram applications in a browser by creating and manipulating operators and connections. Highcharts\footnote{\url{http://www.highcharts.com}} library is utilized to create interactive charts for our application. In the server side, we used a web server suitable for data-intensive real-time applications. Therefore, node.js and its supporting modules such as nodemailer and socket.io were employed. AutoStudio prominently uses open source softwares and frameworks. The client side, including HTML5, JavaScript, Cascading Style Sheets (CSS), jQuery (and helping libraries), twitter bootstrap, and hogan.js, is used for building the graphical user interface, performing Ajax requests, file uploading and downloading, etc. AutoStudio extensively uses pre-compiled hogan templates where the data returned from the server is simply passed to these templates for quick rendering. In addition, Draw2D touch is used to enable creation of diagram applications in a browser by creating and manipulating operators and connections. Highcharts\footnote{\url{http://www.highcharts.com}} library is utilized to create interactive charts for our application. In the server side, we used a web server suitable for data-intensive real-time applications. Therefore, node.js and its supporting modules such as nodemailer and socket.io were employed.
......
...@@ -34,7 +34,7 @@ ...@@ -34,7 +34,7 @@
%\crdata{0-12345-67-8/90/01} % Allows default copyright data (0-89791-88-6/97/05) to be over-ridden - IF NEED BE. %\crdata{0-12345-67-8/90/01} % Allows default copyright data (0-89791-88-6/97/05) to be over-ridden - IF NEED BE.
% --- End of Author Metadata --- % --- End of Author Metadata ---
\title{The PipeFlow Approach: Write once, Run in Different Stream-processing Engines} \title{The PipeFlow Approach: Write Once, Run in Different Stream-processing Engines}
\numberofauthors{2} % in this sample file, there are *total* \numberofauthors{2} % in this sample file, there are *total*
% of EIGHT authors. SIX appear on the 'first-page' (for formatting % of EIGHT authors. SIX appear on the 'first-page' (for formatting
% reasons) and the remaining two appear in the \additionalauthors section. % reasons) and the remaining two appear in the \additionalauthors section.
......
...@@ -61,7 +61,7 @@ where \texttt{limit} parameter limits the number of results returned. ...@@ -61,7 +61,7 @@ where \texttt{limit} parameter limits the number of results returned.
\caption{\PipeFlow architecture} \caption{\PipeFlow architecture}
\label{fig:arch} \label{fig:arch}
\end{figure} \end{figure}
In our approach, we have adopted the automated code translation (ACT) technique by taking an input source script written in \PipeFlow language and converting it into another output program. In general, \PipeFlow system is written in Java and depends heavily on ANTLR\footnote{\url{http://www.antlr.org}} and StringTemplate\footnote{\url{http://www.stringtemplate.org}} libraries. The former generates a \PipeFlow language parser that can build and walk parse trees whereas the latter generates code using pre-defined templates. Basically, the following components are used to achieve the translation of \PipeFlow source script to equivalent target programs (PipeFabric, Spark Streaming, or Storm): \emph{(1)} parser \emph{(2)} flow graph \emph{(3)} template file and \emph{(4)} code generator. The latter two components are specific to target programs and differ from each other depending on the target code to be generated. Therefore, for every target code a separate template file and code generator are created. In our approach, we have adopted the automated code translation (ACT) technique by taking an input source script written in \PipeFlow language and converting it into another output program. In general, \PipeFlow system is written in Java and depends heavily on ANTLR\footnote{\url{http://www.antlr.org}} and StringTemplate\footnote{\url{http://www.stringtemplate.org}} libraries. The former generates a \PipeFlow language parser that can build and walk parse trees whereas the latter generates code using pre-defined templates. Basically, the following components are used to achieve the translation of \PipeFlow source script to equivalent target programs (PipeFabric, Spark Streaming, or Storm): \emph{(1)} parser \emph{(2)} flow graph \emph{(3)} template file and \emph{(4)} code generator. The latter two components are specific to target programs (i.e., engine) and differ from each other depending on the target code to be generated. Therefore, for every target code a separate template file and code generator are created.
\textbf{Role of Components}: The roles and functionalities of each above-mentioned components are described below and shown in Fig. \ref{fig:arch}. \textbf{Role of Components}: The roles and functionalities of each above-mentioned components are described below and shown in Fig. \ref{fig:arch}.
\begin{description} \begin{description}
...@@ -81,7 +81,7 @@ Consider below a simple\footnote{Because of the limitation in the number of page ...@@ -81,7 +81,7 @@ Consider below a simple\footnote{Because of the limitation in the number of page
\$2 := aggregate(\$1) on x \$2 := aggregate(\$1) on x
generate x, sum(y) as counter; generate x, sum(y) as counter;
\end{alltt} \end{alltt}
The following represents the most important parts in Storm and PipeFabric codes, respectively. Both codes have the same functionalities but in different syntax and engines. The following represents the most important parts in Storm and PipeFabric codes, respectively. The \PipeFlow script and both codes have the same functionalities but in different structures and syntaxes.
\begin{lstlisting}[caption=Generated Storm code, label="storm"] \begin{lstlisting}[caption=Generated Storm code, label="storm"]
public class MyFilter extends BaseFilter { public class MyFilter extends BaseFilter {
public boolean isKeep(TridentTuple tuple) { public boolean isKeep(TridentTuple tuple) {
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment