This tutorial series will show you, how you can turn an Xtext project into a web application.

In part 2 of this series we have created a working Vue.js replica of the auto generated Xtext web server. Now the interesting part begins: we will export the code generator to compile the code editor input to JavaScript to run it in the browser. Furthermore, we will create a console component to print some output, generated by the translated code.

What we will do in this part

Take a look at the git repository, if you get stuck:

ngreve/xtext_web
Contribute to ngreve/xtext_web development by creating an account on GitHub.

How This Works

When you get lost during this tutorial, use this flow chart to see what we are trying to achieve:

Flow chart about running your DSL in the browser

The Code Generator

During this tutorial I will use the terms "code generator" and "compiler" interchangeably.

In this section we will modify the *.MWE2 file, so we are able to export the compiler as a JAR file, like we did with the language server. First we will prepare the compiler for the export process, so you can get an idea why and how things are implemented the way they are. In the end I will tell you, what you have to do to run your code in the browser. I will not explain all my grammar rules or how the code generator handles them. I will only explain the broader idea what you have to do, to run your DSL in the browser. The most crucial concepts will be explained by screenshot of the relevant code snippets.

Before we can export the code generator like we did with the language server, we have to edit the *.MWE2 file, so we can export the code generator as a runnable JAR file.

MWE2 file, edited to be able to export the code generator 

The image above shows, how you have to edit the *.MWE2 file.

Within the language section add the generator section with the generateXtendMain parameter set to true. Execute the MWE2 workflow afterwards.

Let's check if everything went well. You should see a Main.xtend file in the *.generator directory. Right click on Main.xtend -> Run As -> Java Application.

Take a look at the Eclipse console. It should display the following error:

Aborting: no path to EMF resource provided!

This is expected behavior, because we didn't provide any input to the code generator.

Open the *.generator/Main.xtend file. You will see a function called runGenerator:

*.generator/Main.xtend

On the fileAccess object you can define the directory in which the compiled code will be stored. src-gen/ is the default path and I will leave it as it is.

For the ExpressionsGenerator class (in *.generator/ExpressionsGenerator.xtend) I will show you the two most crucial points.

Within the doGenerate function of my code generator, I save the generated code to a file named generated_code.js.

*.generator/ExpressionsGenerator.xtend

Remember that the Main.xtend defines that the code will be stored in the src-gen directory next to the compiler. Furthermore, the ExpressionsGenerator class defines that the code will be stored in a file named generated_code.js. So if we later run the compiler JAR file, we will find our compiled code in src-gen/generated_code.js, next to the JAR file itself. This is important to remember, because later we have to be able to handle the compiler, its needed files and the generated files in the correct way, to send the generated code back to the user.

The goal of this part is to implement a console component to show some output. So we need a command to tell the frontend that it has to print something. My rule for that is named PrintCommand. The rule looks like this:

And it will be translated like so:

ExpressionsGenerator.xtend

For example ...

var a = 4
print a

... will be compiled to:

var a = 4
window.printConsoleOutput((a))

Yes, for the sake of simplicity, the variable declaration stays actually the same.
Every grammar rule has to be compiled to its JavaScript equivalent.
The shown printConsoleOutput function will be defined later in this tutorial, when we implement the ConsoleView component for the frontend.

If your language needs to communicate with the browser in any way, define the necessary function on the window object in the frontend and use it in your compiled code.

The concept is to prepare the environment in the frontend and insert the compiled DSL code, which is just JavaScript, into the HTML document.

You will see what I actually mean by that, when we will implement the ConsoleView, this and more we will do in the next chapter.

Frontend

What we will do now, is to implement a module with which we can communicate with the backend. This module will perform an HTTP request, to send the DSL code to the backend where it gets compiled to JavaScript. Afterwards we will create the ConsoleView component with which we can display messages. In the end we will bring all the new functionalities together in the App.vue file where we can run our program by hitting the Run button, which we will also implement.

Sending Requests to the Backend

The communication with the backend is realized by sending HTTP request to the server. We will use the axios package for that:

$ cd <project>/frontend/
$ npm install --save axios
Install the axios package for the frontend

Prepare two files:

frontend
├── . . . 
└── src
    ├── . . .
    └── services
        ├── Api.js <-- Create this
        ├── CompilerService.js <-- Create this
        └── ConnectionData.js
  • Api.js which provides the axios object to send HTTP requests.
  • CompilerServices.js which is responsible for everything that is "compiling" related.

The Api module prepares and returns the axios object:

// FILE: <project>/frontend/src/services/Api.js

import axios from 'axios'
import { protocol, baseUrl } from './ConnectionData.js'

export default () => {
  return axios.create({
    baseURL: protocol + baseUrl
  })
}
<project>/frontend/src/services/Api.js

The CompilerService module looks like this:

// FILE: <project>/frontend/src/services/CompilerService.js

import Api from '@/services/Api'

export default {
  compileCode (dslCode) {
    return Api().post('compile', {
      code: dslCode
    })
  }
}
<project>/frontend/src/services/CompilerService.js

The compileCode function takes the dslCode as an argument. It sends the dslCode via a post request to the /compile path to the backend, using the prior created Api module.

Creating the Console Component

We need something to output the result of our calculations. The simplest way is to visualize the results using text output. Create the ConsoleView.vue file:

./frontend/
| . . .
└── src
    ├── App.vue
    ├── assets
    ├── components
    |   └── ConsoleView.vue <-- Create this
    ├── main.js
    └── services
        

If you still have the HelloWorld.vue component in there, you obviously can delete that.

The ConsoleView component is implemented like so:

<template>
  <!-- FILE: <project>/frontend/src/components/ConsoleView.vue -->
  <div class="console">
    <div class="console_header">
      <div class="console_settings">
        <i
          class="fa fa-trash"
          title="Clear console"
          @click="messages = []"
        />
        <div class="autoscroll" title="Always scroll to last console output">
          <small>Autoscroll:</small>
          <input type="checkbox" v-model="autoscroll"/>
        </div>
      </div>
      <div class="console_header_filler">
        <small class="console_title">Console</small>
      </div>
    </div>
    <pre id="console-output">
        <!-- eslint-disable-next-line -->
        <code v-for="msg in messages"><span class="timestamp">[{{msg.time}}]:&nbsp;</span>{{ msg.message }}<br></code>
    </pre>
  </div>
</template>

<script>

export default {
  name: 'ConsoleView',
  data () {
    return {
      autoscroll: true,
      consoleOutputDOM: null,
      messages: []
    }
  },
  mounted () {
    this.consoleOutputDOM = document.getElementById('console-output')
    window.printConsoleOutput = this.printConsoleOutput
  },
  methods: {
    printConsoleOutput (msg) {
      const d = new Date()
      const sec = (d.getSeconds() <= 9) ? '0' + d.getSeconds() : d.getSeconds()
      const mins = (d.getMinutes() <= 9) ? '0' + d.getMinutes() : d.getMinutes()
      const hours = (d.getHours() <= 9) ? '0' + d.getHours() : d.getHours()
      const month = (d.getMonth() + 1 <= 9) ? '0' + (d.getMonth() + 1) : (d.getMonth() + 1)
      const date = (d.getDate() <= 9) ? '0' + d.getDate() : d.getDate()
      const year = d.getYear() + 1900
      this.messages.push({
        time: `${month}/${date}/${year} ${hours}-${mins}-${sec}`,
        message: msg
      })
      if (this.autoscroll) {
        // Prevent blocking and wait for the message to render
        setTimeout(() => {
          this.scrollToBottom()
        }, 1)
      }
    },
    scrollToBottom () {
      try {
        this.consoleOutputDOM.lastElementChild.scrollIntoView()
      } catch (err) {
        console.log('ConsoleView - scrollToBottom: ' + err)
      }
    }
  }
}

</script>

<style>
  @import url('https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css');

  :root {
    --light-grey: #E0E0E0;
    --grey: #a0a0a0;
  }

  i {
    cursor: pointer;
    height: 1em;
    width: 1em;
  }

  #console-output {
    flex-grow: 1;
    padding-left: 0.5em;
  }

  #console-output code {
    white-space: pre-wrap;
    overflow: auto;
  }

  .autoscroll {
    display: flex;
    flex-direction: row;
    align-items: center;
  }

  .timestamp {
    font-weight: bold;
    color: var(--grey);
  }

  pre {
    white-space: pre-line;
    margin: 0;
    margin-top: -1em;
  }

  .console_settings {
    flex-grow: 1;
    flex-wrap: nowrap;
    flex-direction: row;
    display: flex;
    align-items: center;
  }

  .console_settings small {
    padding-top: 3px;
  }

  .console_settings > * {
    margin-left: 0.5em;
  }

  .console_header {
    padding-left: 0.5em;
    position: sticky;
    top: 0;
    left: 0;
    right: 0;
    height: 1.3em;
    display: flex;
    flex-direction: row;
    align-items: center;
    background-color: var(--light-grey);
    line-height: 1em;
  }

  .console_title {
    padding-top: 3px;
    color: var(--grey);
  }

  .console_header_filler {
    display: flex;
    flex-grow: 1;
    padding-right: 3em;
    justify-content: flex-end;
    flex-direction: row;
  }

  .console {
    display: flex;
    flex-direction: column;
    overflow-y: scroll;
  }
</style>
<project>/frontend/src/components/ConsoleView.vue

As soon as the ConsoleView component is mounted, it registers its printConsoleOutput(...) function on the window object, which makes it available everywhere in the web application. This function is used by the print command of the DSL. It will be used by the compiled DSL code to send messages to the console component.

The printConsoleOutput(...) function fills the messages array of the ConsoleView component with the message it has to display.
The objects within the messages array contain a timestamp and the message.

Furthermore, the component is capable of automatically scrolling to the last printed message and clearing all the messages by clicking the trash icon. Auto scroll is done within the scrollToBottom(...) function.

Importing the New Functionalities

In the last step we will have to extend the App.vue component. We will add the ConsoleView component, and we will implement a button which triggers a function which sends the code editor content to the backend, where it will be compiled to JavaScript.

The new App.vue looks like this:

<template>
  <!-- FILE: <project>/frontend/src/App.vue -->
  <div id="app">
    <div id="controls">
      <!-- Click on "Run" triggers the compiling process -->
      <button
        @click="compileToJS"
      >
        Run
      </button>
    </div>
    <div class="content">
      <div id="xtext-editor" :data-editor-xtext-lang="this.dslFileExtenstion"></div>
      <ConsoleView id="console-view" />
    </div>
  </div>
</template>

<script>
import { protocol, baseUrl } from '@/services/ConnectionData.js'
import ConsoleView from '@/components/ConsoleView'
import CompilerService from '@/services/CompilerService'

export default {
  name: 'App',
  components: {
      ConsoleView
  },
  data () {
    return {
      xtextEditor: null,
      scriptContainer: null,
      dslFileExtenstion: ''
    }
  },
  mounted () {
    /* If the _xtext object is not null when we mount this component, we can continue to configure our editor, otherwise we will wait for the 'ready' event */
    (!window._xtext) ? window.xtextReadyEvent.on('ready', this.setXtextEditor) : this.setXtextEditor()
  },
  methods: {
    setXtextEditor () {
      /* The serviceUrl contains the URL, on which
         the language server is reachable */
      this.dslFileExtenstion = window._dslFileExtenstion

      /* We have to wait untill rendering of this.dslFileExtenstion
        in data-editor-xtext-lang attribute finishes
        before we initialize the editor */
      this.$nextTick(() => {
        this.xtextEditor = window._xtext.createEditor({
          baseUrl: '/',
          serviceUrl: `${protocol}${baseUrl}xtext-service`,
          syntaxDefinition: `xtext-resources/generated/mode-${this.dslFileExtenstion}.js`,
          enableCors: true
        })
      })
    },
    async compileToJS () {
      try {
        // Take the editor content and send it to the backend
        const response = await CompilerService.compileCode(this.xtextEditor.getValue())
        const compiledCode = response.data.code
        // Execute the compiled code
        this.runCompiledCode(compiledCode)
      } catch (err) {
        console.error(err.error)
      }
    },
    runCompiledCode (compiledCode) {
      try {
        const headDOM = document.getElementsByTagName('head')[0]
        // Remove prior created script DOM if one exists
        if (this.scriptContainer)
          headDOM.removeChild(this.scriptContainer);

        // Create a script DOM which will contain the compiled code
        this.scriptContainer = document.createElement('script')
        this.scriptContainer.innerHTML = `
        try {
          async function run() {
              ${compiledCode}
          }
          run()
        } catch (err) {
          console.error('COMPILED CODE ERROR:', err)
        }
        `
        // Appending the script DOM to the head will lead to autmatic execution
        // of the appended script DOM
        headDOM.appendChild(this.scriptContainer)
      } catch (err) {
        console.error(err)
      }
    }
  }
}
</script>

<style>
html
, body
, #app {
  height: 100%;
  margin: 0;
  padding: 0;
}

body {
  align-items: center;
  justify-content: center;
  display: flex;
  width: 100%;
}

#app {
  display: flex;
  width: 100%;
  flex-direction: column;
  font-family: Avenir, Helvetica, Arial, sans-serif;
  -webkit-font-smoothing: antialiased;
  -moz-osx-font-smoothing: grayscale;
  color: #2c3e50;
  height: 90%;
  justify-content: center;
  align-items: center;
}

#controls,
.content {
  width: 90%;
}

.content {
  display: flex;
  min-height: 100%;
}

#xtext-editor {
  position: relative;
  height: 100%;
  width: 100%;
  border: 1px solid #aaa;
}

#console-view {
  position: relative;
  width: 100%;
  height: 100%;
  border: 1px solid #aaa;
}
</style>
<project>/frontend/src/App.vue

In the <template> section we include the prior created ConsoleView component and add the Run button, which triggers the compileToJs(...) function.

The new functions are:

  • compileToJs(...): This function gets the Xtext editor content and uses the CompilerService to send the DSL code to the backend. As soon as this functions receives a response it takes the compiled code and calls the runCompiledCode(...) function.
  • runCompiledCode(...): This function creates a script DOM and inserts the compiled code. The created script DOM will be added to the head DOM. As soon as it's added to the document, it will be executed automatically.

The frontend is now ready to display some output. But to actually display messages, it needs the compiled code. Let's continue with the compiler and the backend.

Export the Compiler

Before we continue with the backend we have to get the compiler from the Xtext project. We have to do this in a similar way as we did with the language server in part 2. The backend will be able to use the exported JAR file to compile the DSL code, which then will be sent back to the user.

First we create the export destination for the compiler:

./backend/
├── compiler <-- Create this directory
├── language_server
├── node_modules
├── package.json
├── package-lock.json
└── src

If you remember the export process of the language server, this workflow will seem very familiar to you. Right click on the *.generator/Main.xtend, select Export... and follow these steps:

In the second step of the export wizard, make the following selections:

  • For Launch configuation select the Main of your DSL project.
  • For the Export destination select the prior created directory.
  • Unlike in the language server export process, select Package required libraries into generated JAR for Library handling.

Click Finish to export the compiler as a runnable JAR file.

If you now navigate to the exported JAR file and run it via the CLI, you should see the same error message like you have seen in Eclipse:

$ cd <Project>/backend/compiler/
$ java -jar ./dsl_compiler.jar
Aborting: no path to EMF resource provided!
Test the compiler

Our command produces an error because the compiler expects the path of a file which contains the DSL source code as it's first argument. Our backend will take care of this ...

Backend

We have to implement the following functionality:

  1. The backend receives the "compile this code" request from the frontend. This request includes the DSL code from the code editor.
  2. The backend then uses the compiler JAR file to compile the given DSL code.
  3. First it will create a temporary working directory in which it will create a file which contains the given DSL code.
  4. It feeds the file with the DSL code into the compiler JAR file which then generates the output to <tmp_working_dir>/src-gen/generated_code.js.
  5. The backend then reads the generated_code.js and sends its content back to the user.
  6. Afterwards the temporary working directory gets deleted.

Let's tell the backend where it can find the compiler. We've stored the compiler at <project>/backend/compiler/dsl_compiler.jar. Furthermore, we will define our language file extension (change this to match your file extension). To have both of this available everywhere in our project we have to extend the config.js:

// FILE: <project>/backend/src/config/index.js
'use-strict'

module.exports = {
  dslFileExt: 'exp', // Add and change this
  paths: {
    dslCompiler: './compiler/dsl_compiler.jar', // Add this
    languageServer: './language_server/language_server.jar'
  }
}
<project>/backend/src/config/index.js

Next we will implement the CodeGeneratorController which is (unsurprisingly) responsible for everything that is related to "code generation". Create the following files and directories:

./backend/
├── compiler
├── language_server
├── package.json
├── package-lock.json
└── src
    ├── app.js
    ├── config
    │   └── index.js
    ├── controllers <-- Create
    │   └── CodeGeneratorController.js <-- Create 
    ├── routes.js
    └── services

As explained in the beginning, the backend creates a temporary working directory in which the compiler JAR can do whatever it needs to do. The name of this working directory will be randomly generated by the uuid package. Install the uuid package:

$ cd <project>/backend
$ npm install --save uuid

Below you can see the CodeGeneratorController. As already explained, it takes the DSL code from the request (from req.body.code), creates a temporary directory and a file and compiles the file using the dsl_compiler.jar.

// FILE: <project>/backend/src/controllers/CodeGeneratorController.js
'use strict'

const config = require('../config')
const fsPromises = require('fs').promises
const path = require('path')
const { execSync } = require('child_process')
const uuidv4 = require('uuid').v4 // random UUID

module.exports = {
  async compileToJS (req, res) {
    /*
     * We prepare a temporary directory to use it as our working directory.
     * The name of this directory will be a automatic generated UUID.
     * The directory will be deleted after we've send the compiled code to the
     * user.
     * It is very unlikely that the  directory names will collide.
     */
    const tmpDir = path.resolve(path.dirname(config.paths.dslCompiler), uuidv4())
    try {
      if (!req.body.code) throw new Error('No code provided')
      // Create the temporary directory
      await fsPromises.mkdir(tmpDir)
      // Define where our dsl code will be at.
      const dslCodePath = path.resolve(tmpDir, `code.${config.dslFileExt}`)
      // Write the sent code to a file
      await fsPromises.writeFile(dslCodePath, req.body.code)
      // Run the compile: the argument is the path to the received code, which
      // we've wrote to a file. with cwd we tell the shell to use the tmpDir
      // As the working directory.
      const dslCompilerPath = path.resolve(process.cwd(), config.paths.dslCompiler)
      execSync(`java -jar ${dslCompilerPath} ${dslCodePath}`, { cwd: tmpDir })
      // After the execution of the command, the compiled code will be available
      // within <tmpDir>/src-gen/generated_code.js
      const generatedPath = path.resolve(tmpDir, 'src-gen', 'generated_code.js')
      // Read the generated JavaScript code ...
      const jsCode = await fsPromises.readFile(generatedPath, 'utf-8')
      // ... and send it back to the user
      await res.send({
        code: jsCode
      })
    } catch (err) {
      console.log(err)
      res.status(500).send({
        error: err
      })
    }
    // Clean up the directory ...
    await fsPromises.rmdir(tmpDir, { recursive: true })
  }
}
<project>/backend/src/controllers/CodeGeneratorController.js

The CodeGeneratorController takes the received DSL code, writes it to a file and feeds the compiler JAR file with it, which then converts this code to JavaScript code. We send this JavaScript code back to the user, where it gets executed in the browser.

Next we have to extend the routes.js so the backend can handle a POST request on the /compile/ path. If you have forgotten why, take a look at the CompilerService module of the frontend.

Extend the routes.js of the backend:

// FILE: <project>/backend/src/routes.js
'use strict'

const httpProxy = require('http-proxy')
const apiProxy = httpProxy.createProxyServer()
const CodeGeneratorController = require('./controllers/CodeGeneratorController') // Add this

// location of our exported language server
const LANGUAGE_SERVER = 'http://localhost:8090/'

/*
 * When an error with our proxy set up occures, we have to handle it
 * in some way. Otherwise our server will crash.
 * In this case we just print the error, and forward it to the client.
 */
apiProxy.on('error', function (error, req, res) {
  console.error('Proxy error:', error)
  if (!res.headersSent) {
    res.writeHead(500, { 'content-type': 'application/json' })
  }

  const json = { error: 'proxy_error', reason: error.message }
  res.end(JSON.stringify(json))
})

module.exports = (app) => {
  // Forward all types of HTTP requests to the LANGUAGE_SERVER
  app.all('/xtext-service/*', function (req, res) {
    apiProxy.web(req, res, { target: LANGUAGE_SERVER })
  })

  app.post('/compile',
    CodeGeneratorController.compileToJS) // Add this
}
Extend routes.js to handle post requests on the /compile path

In the routes.js we tell the backend that if it receives a post request on the /compile path, it should be managed by the CodeGeneratorController.compileToJS(...) function.

The backend is now able to handle the request triggered by the Run button of your frontend. It will take the given code, compile it to JavaScript and send it back to the user.

Nodemon and the Code Generator

The CodeGeneratorController is generating output files, because of Nodemon this would trigger the restart of the backend. Like we did with the language_server/ directory, we also have to add the compiler/ to the list of directories, which should be ignored by Nodemon.

Open the <project>/backend/package.json and add the following nodemonConfig section:

{
  "name": "backend",
  . . .
  "scripts": {
    . . .
  },
  "nodemonConfig": {
    "ignore": [
      "language_server/",
      "compiler/"
    ]
  },
  . . .
}

Afterwards close and restart your Node backend to reload these configurations.

Once more, we now can bring it all back together.

Run Your Code

If you now start your backend and frontend, your web application should be able to compile and run the code, as already shown in the beginning:

Conclusion Part 3

Our frontend sends the content of the code generator to the backend where it gets compiled to JavaScript code. The compiled code is sent back to the frontend where it gets injected in the HTML document where it runs. Why does this work? The frontend provides the necessary environment with functions used by the DSL, like in my case window.printConsoleOutput(...).

Every web functionality you want to implement is achievable by this concept. Your code generator just has to compile your DSL data structures and functionalities to their JavaScript equivalents.

What's next?

If you want to improve the shown web application, here are some ideas:

  • Disable the Run button while a compiling  request is pending.
  • Display errors in the console. For example if you try to send a compiling request with an empty editor, display "No code provided" in the console.

I will leave this application as it is, because I just wanted to show the overall concept, which, I hope, should be clear from now on.

In the next and last part, I will explain how to make your DSL web application available to the world, by setting up a NGINX server as a reverse proxy.

See you in part 4!