Xtext on the Web

This tutorial series will show you, how you can turn an Xtext project into a web application.

In part 1 of this series we've created an Xtext project with web integration capabilities. In this part we will take this Xtext web application and turn it into a Vue.js application.

Prerequisites

To start the journey, you have to install some additional software:

Node.js
wget command line tool (Windows installation guide)

Furthermore, you should know, that I am using Linux. If you are using Windows, you should take care of any paths used in this Tutorial. Convert them accordingly.

Take a look at the git repository if you get stuck:

Starting With the Frontend

In this chapter I will explain how to set up a new Vue.js project. In the next section I will port the Xtext editor from Eclipse into the newly created Vue project.

If you have no clue what is happening here and you want to learn Vue.js, I recommend searching for a tutorial on YouTube. I've learned it with this tutorial series from freeCodeCamp.org by Cody Seibert. This series creates a full stack web app, including database functionalities. If you complete this tutorial, you definitely know how front- and backend interaction works.

Install vue-cli globally by typing sudo npm install -g @vue/cli @vue/cli-service-global in the command line.

Create an empty project directory. This will, in a second, contain our frontend and later our backend too. Switch into the freshly created directory and run vue create frontend (the last parameter is the project name). This will open the project wizard:

$ cd <project>
$ vue create frontend

Vue CLI v4.2.2
? Please pick a preset: (Use arrow keys)
❯ default (babel, eslint) 
  Manually select features

? Pick the package manager to use when installing dependencies: 
  Use Yarn 
❯ Use NPM

For the preset select: default (babel, eslint)
For the package manager select: Use NPM

Your project directory should now look like this:

<project>/
└── frontend
    ├── babel.config.js
    ├── node_modules
    ├── package.json
    ├── package-lock.json
    ├── public
    ├── README.md
    └── src

To run the auto generated Vue project, switch into the frontend directory and enter.npm run serve The command line will tell you, where your frontend is hosted.

$ cd expressions_web_dsl/frontend
$ npm run serve

App running at:
- Local:   http://localhost:8080/ 
- Network: http://192.168.178.38:8080/

Note that the development build is not optimized.
To create a production build, run npm run build.

Visit http://localhost:8080 to check if everything went well so far. You should see the Vue.js default web app.

This tutorial will only give you an overview of the most important components. I encourage you, to take a look at the git repository - branch part_2 - if you get stuck. I placed some helpful comments in there.

The Code Editor

Vue.js uses components to build the frontend. The only component you will need so far, is the code editor. In the next steps we will get all the necessary resources from the auto generated Xtext web server, to build such a component.

Get the Libraries

The *.web/ directory of the Xtext project shows us some of the resources to run the code editor, but not all of them. As soon as you start the Xtext web server, the web server will use web jars, which are web libraries as JAR Files. Start your Xtext web server, as already shown in part 1, and open http://localhost:8080/webjars/ to have a look at some of the libraries, which are used in the background. Make sure to close the Vue process first, so the used ports do not collide.

To get the code editor up and running in the Vue project, we have to extract these libraries.

In the next step we will put these libraries in the public directory of our Vue project.

<project>/
└── frontend
    ├── babel.config.js
    ├── node_modules
    ├── package.json
    ├── package-lock.json
    ├── public		<----- destination dir
    ├── README.md
    └── src

One way to get these libraries is to use the wget command line tool.

With the Xtext web server running, open the terminal and enter the following command.

wget --mirror --reject=index.html,style.css -nH -P public/ http://localhost:8080/{xtext,xtext-resources,webjars}

On Microsoft Windows it seems wget behaves a bit differently. Windows users should try:

wget -r -nH -np --reject=index.html,style.css -P public/ http://localhost:8080/{xtext,xtext-resources,webjars}

From wget --help:

--mirror shortcut for -N -r -l inf --no-remove-listing (mirrors the given URL)
--reject comma-separated list of rejected extensions
-nH don't create host directories
-P save files to PREFIX/..
http://localhost:8080 is the address of your Xtext web server

This command will recursively download three directories into your <project>/frontend/public directory of your Vue.js project: xtext, webjars and xtext-resources:

frontend/
├── babel.config.js
├── node_modules
├── package.json
├── package-lock.json
├── public
│   ├── webjars
│   │   ├── ace
│   │   ├── jquery
│   │   └── requirejs
│   ├── xtext
│   │   └── 2.18.0.M3 <-- Version numbers can be different
│   └── xtext-resources
│       └── generated
├── README.md
└── src

Extracted resources (only relevant files are listed)

When you change the grammar of your DSL, you have to delete the downloaded xtext-resources directory and rerun the command, to update syntax highlighting within the code editor.

Creating the Editor

To create a first result, we have to edit/create three files. First we will create a little helper class, to create a communication channel between the index.html and the App.vue.

We start by stealing some code from Alex Taujenis (thanks!). Create <project>/frontend/public/Events.js with the following content:

/*
 * FILE: <project>/frontend/public/Events.js
 * Stolen from Alex Taujenis: https://gist.github.com/alextaujenis/0dc81cf4d56513657f685a22bf74893d
 * MIT license
 */

class Events {
  constructor () {
    this._callbacks = {}
  }

  on (key, callback) {
    // create an empty array for the event key
    if (this._callbacks[key] === undefined) { this._callbacks[key] = [] }
    // save the callback in the array for the event key
    this._callbacks[key].push(callback)
  }

  emit (key, ...params) {
    // if the key exists
    if (this._callbacks[key] !== undefined) {
      // iterate through the callbacks for the event key
      for (let i=0; i<this._callbacks[key].length; i++) {
        // trigger the callbacks with all provided params
        this._callbacks[key][i](...params)
      }
    }
  }
}

<project>/frontend/public/Events.js

We continue with the <project>/frontend/public/index.html of the Vue.js project. Below you can see the result after the editing:

<!DOCTYPE html>
<!-- FILE: <project>/frontend/public/index.html -->
<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width,initial-scale=1.0">
    <link rel="icon" href="<%= BASE_URL %>favicon.ico">
    <title><%= htmlWebpackPlugin.options.title %></title>

    <!-- Class to notify App.vue that
         the code editor is ready to use -->
    <script src="<%= BASE_URL %>Events.js"></script>

    <!-- In principle copied from *.web/index.html -->
    <link rel="stylesheet" type="text/css" href="<%= BASE_URL %>xtext/<%= VUE_APP_XTEXT_ACE %>/xtext-ace.css"/>
    <script src="<%= BASE_URL %>webjars/requirejs/<%= VUE_APP_REQUIRE_JS %>/require.min.js"></script>

    <script type="text/javascript">
      var xtextReadyEvent = new Events()
      var baseUrl = '<%= BASE_URL %>';
      var fileIndex = baseUrl.indexOf('index.html');

      /* _xtext will contain our ready-to-use editor. The editor object will
         be available through the window object. */
      var _xtext = null;
      var _dslFileExtenstion = '<%= VUE_APP_DSL_FILE_EXT %>';

      if (fileIndex > 0)
        baseUrl = baseUrl.slice(0, fileIndex);
      require.config({
        baseUrl: baseUrl,
        paths: {
          'jquery': 'webjars/jquery/<%= VUE_APP_JQUERY %>/jquery.min',
          'ace/ext/language_tools': 'webjars/ace/<%= VUE_APP_ACE %>/src/ext-language_tools',
          'xtext/xtext-ace': 'xtext/<%= VUE_APP_XTEXT_ACE %>/xtext-ace'
        }
      });
      require(['webjars/ace/<%= VUE_APP_ACE %>/src/ace'], function() {
        require(['xtext/xtext-ace'], function(xtext) {
          _xtext = xtext;
          /* The editor (_xtext) is now ready-to-use.
             We emit an 'ready' event to inform the App.vue component about it. */
          xtextReadyEvent.emit('ready');
          /* The information flow continues in the mounted function of the
             App.vue component. */
        })
      })
    </script>

  </head>
  <body>
    <noscript>
      <strong>We're sorry but <%= htmlWebpackPlugin.options.title %> doesn't work properly without JavaScript enabled. Please enable it to continue.</strong>
    </noscript>
    <div id="app"></div>
    <!-- built files will be auto injected -->
  </body>
</html>

<vue_project>/public/index.html

Maybe you are wondering what these <%= VARIABLE %> are. Vue.Js is using Webpack to build the project. And these kinds of variables are Webpack placeholders. If you want to learn more about environment variables in Vue.Js, I recommend reading the official documentation.

We define the value of these placeholders in <project>/frontend/.env. Create the .env file:

VUE_APP_ACE=1.3.3
VUE_APP_JQUERY=3.3.1-1
VUE_APP_REQUIRE_JS=2.3.6
VUE_APP_XTEXT_ACE=2.18.0.M3
VUE_APP_DSL_FILE_EXT=exp

<project>/frontend/.env

We use them to define the version numbers of the libraries we've pulled from the Xtext web project. Don't forget to set VUE_APP_DSL_FILE_EXT to match your DSL file extension.

Take a look in the directories of the extracted libraries to figure out which versions you have. These were mine:

frontend/
├── public
│   ├── webjars
│   │   ├── ace
│   │   │   └── 1.3.3 	<--- Version of ace
│   │   ├── jquery
│   │   │   └── 3.3.1-1	<--- Version of jquery
│   │   └── requirejs
│   │       └── 2.3.6 	<--- Version of requirejs
│   ├── xtext
│   │   └── 2.18.0.M3 	<--- Version of xtext(-ace)
│   │       └── images
│   └── xtext-resources
│       └── generated
└── src

Check the library versions (only relevant files are listed)

Before we continue porting the code editor, we have to store the address of our backend somewhere. We don't have a backend at this point, this is just forethought. Create the directory <project>/frontend/src/services and within that directory create ConnectionData.js. This module will contain the URL of the backend:

// FILE: <project>/frontend/src/services/ConnectionData.js

module.exports = {
	baseUrl: 'localhost:8085/',
	protocol: 'http://'
}

<project>/frontend/src/services/ConnectionData.js

Now we can build the editor in the <project>/frontend/App.vue file:

<template>
  <!-- FILE: <project>/frontend/src/App.vue -->
  <div id="app">
    <div class="content">
      <div id="xtext-editor" :data-editor-xtext-lang="this.dslFileExtenstion"></div>
    </div>
  </div>
</template>

<script>
import { protocol, baseUrl } from '@/services/ConnectionData.js'

export default {
  name: 'App',
  data () {
    return {
      xtextEditor: null,
      dslFileExtenstion: ''
    }
  },
  mounted () {
    /* If the _xtext object is not null when we mount this component, we can continue to configure our editor, otherwise we will wait for the 'ready' event */
    (!window._xtext) ? window.xtextReadyEvent.on('ready', this.setXtextEditor) : this.setXtextEditor()
  },
  methods: {
    setXtextEditor () {
      /* The serviceUrl contains the URL, on which
         the language server is reachable */
      this.dslFileExtenstion = window._dslFileExtenstion

      /* We have to wait untill rendering of this.dslFileExtenstion
        in data-editor-xtext-lang attribute finishes
        before we initialize the editor */
      this.$nextTick(() => {
        this.xtextEditor = window._xtext.createEditor({
          baseUrl: '/',
          serviceUrl: `${protocol}${baseUrl}xtext-service`,
          syntaxDefinition: `xtext-resources/generated/mode-${this.dslFileExtenstion}.js`,
          enableCors: true
        })
      })
    }
  }
}
</script>

<style>
html
, body
, #app {
  height: 100%;
}

#app {
  font-family: Avenir, Helvetica, Arial, sans-serif;
  -webkit-font-smoothing: antialiased;
  -moz-osx-font-smoothing: grayscale;
  color: #2c3e50;
}

.content {
  position: relative;
  width: 45%;
  height: 80%;
}

#xtext-editor {
  position: relative;
  height: 100%;
  border: 1px solid #aaa;
}
</style>

<project>/frontend/src/App.vue

We are done with the editor. If you start your Vue.js frontend with npm run serve, you now should see the editor window, we've already had in the *.web/ project. But at this point, the editor does not show any syntax errors. If you open the developer tools of your browser and watch the network communication, you can see what is happening in the background:

Background calls of the code editor while typing

The input is sent to the language server, defined in the window._xtext.createEditor(...) function call in the App.vue file. Unfortunately nobody is answering.

Starting with the Backend

Our frontend is constantly trying to call the (not yet existent) backend. Time to answer these calls, by building one. First we will initialize our backend project. The resulting Node.Js application will act as a reverse proxy for the language server. Later in this series the Node backend will have an actual use case, which is compiling the code. So it must be able to differentiate between two kinds of requests: the one for the language server and the one for itself.

In the next section we will export the language server as a runnable JAR file, with which we are able to run the language server without the need of eclipse. Our backend will take care of the start of the language server JAR file.

Your current project directory should only contain the Vue.js frontend. In the next step create the backend directory next to it:

<project>/
├── backend <--- Create this one
└── frontend
    ├── babel.config.js
    ├── node_modules
    ├── package.json
    ├── package-lock.json
    ├── public
    ├── README.md
    └── src

Navigate into the backend directory and execute the npm init command. Fill out the questions about your project. Set the entry point to app.js:

$ cd backend
$ npm init
This utility will walk you through creating a package.json file.
It only covers the most common items, and tries to guess sensible defaults.

See `npm help json` for definitive documentation on these fields
and exactly what they do.

Use `npm install <pkg>` afterwards to install a package and
save it as a dependency in the package.json file.

Press ^C at any time to quit.
package name: (backend) 
version: (1.0.0) 
description: Backend for a domain specific language
entry point: (index.js) app.js
test command: 
git repository: 
keywords: 
author: Nico Greve
license: (ISC) 
About to write to /home/nico/prog/backend/package.json:

{
  "name": "backend",
  "version": "1.0.0",
  "description": "Backend for a domain specific language",
  "main": "app.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "Nico Greve",
  "license": "ISC"
}


Is this OK? (yes)

Furthermore, we have to install some packages:

# In the Backend directory
$ npm install --save express body-parser cors nodemon eslint morgan http-proxy

After the installation process finishes we have to set up the ESLint config, with the node node_modules/eslint/bin/eslint.js --init command. Answer the questions as shown below:

# In the Backend directory
$ node node_modules/eslint/bin/eslint.js --init

? How would you like to use ESLint? To check syntax, find problems, and enforce code style
? What type of modules does your project use? JavaScript modules (import/export)
? Which framework does your project use? None of these
? Does your project use TypeScript? No
? Where does your code run? Node
? How would you like to define a style for your project? Use a popular style guide
? Which style guide do you want to follow? Standard: https://github.com/standard/standard
? What format do you want your config file to be in? JavaScript
Checking peerDependencies of eslint-config-standard@latest
Local ESLint installation not found.
The config that you've selected requires the following dependencies:

eslint-config-standard@latest eslint@>=6.2.2 eslint-plugin-import@>=2.18.0 eslint-plugin-node@>=9.1.0 eslint-plugin-promise@>=4.2.1 eslint-plugin-standard@>=4.0.0
? Would you like to install them now with npm? Yes

In the last step of preparing the backend, we have to change some configurations in the backend/package.json file. Edit the scripts section in the following way:

  "scripts": {
    "start": "./node_modules/nodemon/bin/nodemon.js src/app.js --exec 'npm run lint && node'",
    "lint": "./node_modules/.bin/eslint src/",
    "test": "echo \"Error: no test specified\" && exit 1"
  },

Code snipped of <project>/backend/package.json

Afterwards prepare your directory in this way:

backend/
├── language_server <-- Create with sub directory
|   └── WebRoot
|       └── index.html <-- empty file
├── node_modules
├── package.json
├── package-lock.json
└── src         <-- Create directories with empty empty files as shown
    ├── app.js
    ├── config
    |   └── index.js
    ├── services
    |   └── LanguageServerService.js
    └── routes.js

In the next steps we will do two things:

We prepare the app.js so our Node.js backend starts for us.
In the routes.js we will tell our backend that it has to redirect all requests, which call the /xtext-service/ path, to the language server. We will export the language server in the next chapter.

Our backend will also control the start of the language server. To do so, it has to know where the language server JAR file is stored. At this point we don't have that JAR file, but we will now define where we will store it: <backend>/language_server/language_server.jar. We will keep that information in src/config/index.js:

// FILE: <project>/backend/src/config/index.js
'use-strict'

module.exports = {
  paths: {
    languageServer: './language_server/language_server.jar'
  }
}

<project>/backend/src/config/index.js

Our backend has now the information where it can access the language server JAR. Next we will build a module to start the JAR file. Implement the LanguageServerService as shown below:

// FILE: <project>/backend/src/services/LanguageServerService.js
'use-strict'

const { spawn } = require('child_process')
const path = require('path')

module.exports = {
  async startLanguageServer (languagePath) {
    return new Promise(function (resolve, reject) {
      const baseDir = path.dirname(path.resolve(languagePath))
      try {
        const ls = spawn(`java -jar ${languagePath}`, {
          shell: true,
          cwd: baseDir
        })

        ls.stdout.on('data', (data) => {
          console.log(`stdout: ${data}`)
        })

        ls.stderr.on('data', (data) => {
          // Waiting for the string that indicates the successfull start
          if (/(.)*INFO(.)*Server started(.)*/.test(data)) {
            resolve()
          }
        })

        ls.on('close', (code) => {
          if (code !== 0) {
            throw new Error(`Error: language server exited with code: ${code}\nHINT: Try to wait ~10 sec before next start. This gives the system the necessary time for it's cleanup processes.`)
          }
        })
      } catch (err) {
        reject(err)
      }
    })
  }
}

<project>/backend/src/services/LanguageServerService.js

Take a look at the spawn(...) function call (Line 12). Depending on how you are going to use your project, I would recommend using the absolute path of your java binary. On Mac and Linux you should be able to figure that out by typing $ which java in the CLI of you system. To use the absolute path is just error prevention, if you plan to auto start your services. Because in some cases there can be problems with the $PATH variable.

The main entry point of the backend process is the src/app.js. In there we will start the backend itself, and we use the prior created LanguageServerService to fire up the language server.
Edit the <project>/backend/src/app.js so it looks like the one down below:

// FILE: <project>/backend/src/app.js
'use strict'
const path = require('path')
const cors = require('cors')
const morgan = require('morgan')
const bodyParser = require('body-parser')
const express = require('express')
const app = express()

const config = require('./config')
const LanguageServerService = require('./services/LanguageServerService.js')

/*
 * To be able to the proxy method for the web editor request,
 * we have to enable Cross-Origin Resource Sharing (CORS).
 * What is CORS: https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS
 */
app.use(cors({
  origin: function (origin, callback) {
    callback(null, origin)
  },
  credentials: true
}))

// Log output format
app.use(morgan('combined'))
// Converts the request and responses automatically into the JSON format
app.use(bodyParser.json())

require('./routes')(app)
async function run () {
  const port = 8085 // Configure your port
  const languageServerPath = path.resolve(process.cwd(), config.paths.languageServer)
  await LanguageServerService.startLanguageServer(languageServerPath)
  app.listen(port) // Start the backend
  console.log(`listining http://localhost:${port}`)
}

run()

<project>/backend/src/app.js

With the src/routes.js we tell the backend how it has to handle different kinds of HTTP requests.

Implement the routes module:

// FILE: <project>/backend/src/routes.js
'use strict'

var httpProxy = require('http-proxy')
var apiProxy = httpProxy.createProxyServer()

// location of our exported language server
const LANGUAGE_SERVER = 'http://localhost:8090/'

/*
 * When an error with our proxy set up occures, we have to handle it
 * in some way. Otherwise our server will crash.
 * In this case we just print the error, and forward it to the client.
 */
apiProxy.on('error', function (error, req, res) {
  console.error('Proxy error:', error)
  if (!res.headersSent) {
    res.writeHead(500, { 'content-type': 'application/json' })
  }

  var json = { error: 'proxy_error', reason: error.message }
  res.end(JSON.stringify(json))
})

module.exports = (app) => {
  // Forward all types of HTTP request to the LANGUAGE_SERVER
  app.all('/xtext-service/*', function (req, res) {
    apiProxy.web(req, res, { target: LANGUAGE_SERVER })
  })
}

<project>/backend/src/routes.js

As you can see, we now redirect all calls which requests the /xtext-service/* path, to the language server, which will be running on http://localhost:8090/.

Let's grab the language server from our Xtext project.

The Language Server and Nodemon

Because it makes life easier, we've set up Nodemon to watch our project and restart the backend as soon as it recognizes file changes. This can cause problems, because the language server JAR file we will export, creates log files, e.g. if an error occurs, which would unnecessarily cause restarts of the backend. Let's tell Nodemon, that changes in the language_server/ can be ignored.

Open the <project>/backend/package.json and add the following nodemonConfig section:

{
  "name": "backend",
  . . .
  "scripts": {
    . . .
  },
  "nodemonConfig": {
    "ignore": [
      "language_server/"
    ]
  },
  . . .
}

Afterwards close and restart your Node backend to reload these configurations.

Export the Language Server

In the previous chapter we configured our Node.js backend in a way, that every /xtext-service/ request gets forwarded to http://localhost:8090 (which is our language server). So we have to change the port the language server is running on, in our Xtext project. Open *.web/src/*.web/ServerLauncher.xtend and change the port:

To Export the language server, right click in the Xtext project on the *.web/ project and select Export....

In the first step select Runnable JAR file, in the second step copy the configuration as shown in the image. Choose the prior created <project>/backend/language_server directory, as the destination path. Start the export process by selecting Finish.

With the terminal navigate to your chosen Export destination and run

java -jar language_server.jar

If everything went well, your terminal should now show you the same messages as the console, when you start the web server within Eclipse.

Bringing It All Together

We reached the state, where we have successfully escaped the eclipse environment and are (more or less) free of choosing the technologies we want to use.

Open two terminals and fire up the Vue.js frontend (top terminal with npm run serve) the Node.js backend (terminal at the bottom with npm run start):

The video shows what to start, and that the language server is now working. The input hello world now causes an error. In the bottom left corner (backend terminal) you can see the language server redirection in action.

Conclusion Part 2

We have learned how web integration in Xtext works and where to configure the backend and frontend.

Furthermore, we have successfully ported the Xtext project into a Vue.js/Node.js project and are now ready to start with a more interesting part: compiling the editor input to JavaScript to run our code in the browser.

See you in Part 3!

Xtext on the Web - Part 2