What is the Command Line Tool and what does it do?

The Command Line Tool (CLI) is a simple program that works in a Unix shell or Windows Terminal. It allows you to create, maintain and deploy Speechly applications on the command line. It provides the same functionalities as the Dashboard, as well as the following additional benefits:

1. The CLI gives you access to some advanced configuration options (most notably Imports and Lookups) that are not available on the Dashboard.
2. Using the CLI allows you to use standard tools to maintain Speechly configurations as part of your other development workflow.

Installation

The tool is open-source and can be found at our Github repository. For easy installation, we also provide binary releases for macOS, Linux and Windows. Once installed, the tool is invoked by typing speechly.

Select your OS for quick installation instructions.

$ brew tap speechly/tap
$ brew install speechly

C:\> scoop bucket add speechly https://github.com/speechly/scoop-bucket
C:\> scoop install speechly

Basic usage and getting help

This is a brief overview of how the CLI works in general. A reference of the most important commands is at the end of this document.

The CLI follows an approach similar to e.g. git or docker where different functionalities of the tool are accessed by specifying a command followed by arguments to this command. The tool provides usage instructions when invoked without sufficient parameters.

For example, invoking

speechly

without other arguments, will print a list of all commands supported by the CLI. Running speechly command will print a brief usage guide for the given command, e.g. typing

speechly create

gives instructions on how to create a new application.

Adding an API token

After installing the CLI, you must obtain an API token from the Speechly Dashboard. The API tokens are project specific. If you need to access multiple projects from the CLI, you must create a separate token for each project.

1. Log in to the Speechly Dashboard
2. Select the project for which you want to create the token for
3. Go to Project Settings → API tokens and create a new token
4. Copy the token and run:

speechly projects add --apikey <api_token>

Now you are ready to start using the CLI!

Using environment variables

Alternatively, you can set the API token as environment variable SPEECHLY_APIKEY. It might be a bit cumbersome working with multiple projects when using an environment variables.

Managing multiple projects

If you have split your applications to multiple projects, you must configure the CLI for each of the projects by following the steps in Adding an API token separately for each project.

You can see a list of all projects that have an API token, as well as the one that is currently active, by invoking

speechly projects

The project that is currently active is indicated by “Current project” in the listing.

Switching between projects is done by invoking

speechly projects use

This allows you to select the desired project from a list of available projects.

Deploying with the CLI

When deploying a Speechly application with the CLI, the application configuration (SAL templates, Entity Data Types, etc) are defined in a YAML formatted file. This Configuration YAML, together with other configuration files if needed, should all be put in the same directory. The contents of this directory are deployed by invoking

speechly deploy <app_id> <directory> -w

where <app_id> should be replaced with the application id you want to deploy. The -w flag means the Command Line Tool will wait until model training is complete.

Note that the CLI builds and uploads a deployment package that contains all files in the specified directory! It is thus highly recommended to store there only files that are relevant to the configuration. In the simplest case this is only the Configuration YAML.

Suggested workflow

1. Create a directory for your application’s configuration.
2. Save files in git or other version control system.
3. Use speechly deploy in your CI pipeline for deploying the configuration.

Updating the API token

Updating the API token of a project in the command line tool is done by modifying the CLI settings file. You can see where the settings file is located by invoking

speechly projects

The first output line should indicate the “Settings file used”. The name of the settings file should be .speechly.yaml and it should be located in your home directory. Open this file in a text editor. You should see something like

contexts:
- name: my_project
  host: api.speechly.com
  apikey: <api_token>
  remotename: My Project
current-context: my_project

where <api_token> is the current API token of project my_project. If you have several projects, they are all shown separately, each with their own API token.

You can generate a new API token by following the steps in Adding an API token. Ensure that you are viewing the appropriate project in the Dashboard before generating the token!

When you have copied the new token, replace <api_token> with the new token. Save the settings file, and the new API token for project my_project is immediately in use.

If you have several projects, you must separately replace the API token for each of these by creating a new token for every project in the Dashboard, and pasting it in the settings file.

Converting an Alexa Model [Beta]

To convert an Alexa Interaction Model to a Speechly configuration

1. copy the Interaction Model JSON from the Alexa Developer Console and save it into a file (e.g. alexa-model.json),
2. enter the directory in which you want the Speechly configuration files to be stored, and
3. and then run

speechly convert /path/to/my-alexa-model.json

Depending on what kind of intents and slots the original Interaction Model had, the directory should now contain a config.yaml file, and possibly a number of .csv files. The config.yaml file contains the configuration, and the .csv files contain imported entity values and lookup mappings. These can be deployed with the deploy command as any other Speechly configuration.

Supported Alexa slot types

The Alexa slot types are mapped to Speechly Standard Variables and Entity Types as follows:

The conversion will fail if the input contains any other Alexa slot type.

Also note that the converter does not support dialogue management features that Alexa skills sometimes rely on. The resulting Speechly application can recognize the intents and slots as they appear in the Alexa Interaction Model, however, Speechly will not trigger follow-up questions for instance if the user’s utterance is missing a required slot.

Evaluate the accuracy of your configuration

Once you have deployed a configuration, it is useful to evaluate its accuracy with a set of test utterances that users of your application might say. Evaluation helps you to verify that these utterances work on your app, and identify those test utterances that are not recognized properly. To use the evaluation feature, you need both the set of test utterances, as well as ground truth annotations (correct intents and entities) for these.

Evaluation consists two steps:

1. Run speechly annotate to submit your test utterances to the API to be annotated.
2. Run speechly evaluate to compute accuracy between the API annotations and the ground truth annotations.

Annotate test utterances with the API

To run this command, you need a text file with a single utterance on each line (without any annotations!). Suppose the file test_utterances.txt contains four test utterances:

can i see blue sneakers
i want two of those
my address is one twenty three imaginary road
delivery tomorrow

Run the following command, where <app_id> is the application id to which you have deployed the configuration you want to test.

speechly annotate -a <app_id> --input test_utterances.txt > annotated_utterances.txt

The resulting annotated_utterances.txt contains the test utterances as annotated by the API in the same order as test_utterances.txt. For example, depending on how the intents and entities are configured, it might contain:

*search can i see [blue|blue](color) [sneakers|category_sneaker](product)
*add_to_cart i want two of those
*set_address my address is [one twenty three imaginary road|123 Imaginary Rd.](address)
*set_delivery_date delivery [tomorrow|2021-10-19](delivery_date)

The annotations are given in a syntax similar to Speechly Annotation Language, with the exception that each entity has two values that are separated by a | character. Of these, the first is the original transcript, while the second is the post-processed value that depends on the Entity data type. If an entity has type string, then both the transcript and post-processed value are equal.

For example, above in the first test utterance, the entity color has the value [blue|blue] meaning that color has type string, while the entity product has the value [sneakers|category_sneaker], meaning that product must have a lookup table to map sneakers to the canonical name category_sneaker. Likewise, the remaining entities, address and delivery_date, have a suitable type associated with them.

Annotate with reference date

The entities with type date let users to utter dates relative to the current date. For example, the utterance “today” normally resolves to the current date. The postprocessed value therefore depends on the day you test the utterance. When you test such utterances, specify a reference_date to fix the resolved dates.

speechly annotate -a <app_id> --reference-date 2021-09-12 --input test_utterances.txt

This command applied to previous example now yields a different annotation for delivery_date that is type date.

...
*set_delivery_date delivery [tomorrow|2021-09-13](delivery_date)
...

Compute accuracy

To compute accuracy, you need both the annotations from the Speechly API, as obtained in previous step, as well as ground truth annotations. The ground truth annotations must be given using the same syntax as the API returns. For example, suppose the file ground_truth.txt contains:

*search can i see [blue|blue](color) [sneakers|category_sneaker](product)
*add_to_cart i want [two|2](amount) of those
*set_address my address is [one twenty three imaginary road|123 Imaginary Rd.](address)
*set_delivery_date delivery [tomorrow|2021-09-12](delivery_date)

These are compared against the API generated annotations by running:

speechly evaluate --input annotated_utterances.txt --ground-truth ground_truth.txt

the command outputs:

Ground truth had:
  *add_to_cart i want [two|2](amount) of those
but prediction was:
  *add_to_cart i want two of those

Ground truth had:
  *set_delivery_date delivery [tomorrow|2021-09-12](delivery_date)
but prediction was:
  *set_delivery_date delivery [tomorrow|2021-10-19](delivery_date)

Matching rows out of total:
2 / 4
Accuracy:
0.50

In the first case the prediction is missing the amount entity. This type of errors can be mitigated by ensuring that the utterance appears in the configuration and is properly annotated. In the second case the delivery_date entity was correctly recognized (from the word “tomorrow”), but since the return value of relative dates resolves to the current day, it is a good idea to make sure all the relative dates in your ground truth data are computed against some fixed reference date, and then use that when running speechly annotate as described above.

If you modify and re-deploy your application, you can run the evaluation again with the same test utterances (test_utterances.txt) and ground truths (ground_truth.txt). Note that you always have to first run speechly evaluate annotate to get the API annotations, and then speechly evaluate accuracy to compare these against the ground truths.

Tip: Prepare ground truth by pre-annotating the test utterances

Creating the ground truth annotations manually from scratch can be tedious. To reduce the amount of work this requires, you can use the annotations generated by speechly evaluate annotate as a starting point. Assuming you already have the non-annotated test utterances in test_utterances.txt, run

speechly annotate -a <app_id> --input test_utterances.txt > ground_truth.txt

Next, go through the content of ground_truth.txt and fix all errors in intents or entities that you find.

Usage in Automation

Fully automated usage is easily possible, you need only the API token and the app_id. As the cli is also published to the Docker hub, it can be added to all tools supporting docker images as run steps.

Basic example with bash, mounting the current directory as the working directory:

export SPEECHLY_APIKEY=your_apikey
export APP_ID=your_appid

# validate app:
docker run -it --rm -e SPEECHLY_APIKEY -v $(pwd):$(pwd) -w $(pwd) speechly/cli validate -a ${APP_ID} config-dir

# deploy app:
docker run -it --rm -e SPEECHLY_APIKEY -v $(pwd):$(pwd) -w $(pwd) speechly/cli deploy -a ${APP_ID} config-dir -w

Github Actions

The configuration validation and deployment tasks can be set up as separate workflows in Github Actions. The following examples expect the app_id and API token to be set up as repository secrets. They also expect the configuration file(s) to be located in configuration-directory in the root of the repository.

Configuration validation

name: validate Speechly config
on:
  pull_request:
    branches:
      - master
    paths:
      - "configuration-directory/**"
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: docker://speechly/cli:latest
        with:
          args: validate -a ${{ secrets.APPID }} configuration-directory
        env:
          SPEECHLY_APIKEY: ${{ secrets.SPEECHLY_APIKEY }}

Configuration deployment

name: deploy Speechly config
on:
  push:
    branches:
      - master
    paths:
      - "configuration-directory/**"
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: docker://speechly/cli:latest
        with:
          args: deploy -a ${{ secrets.APPID }} configuration-directory -w
        env:
          SPEECHLY_APIKEY: ${{ secrets.SPEECHLY_APIKEY }}

CLI Reference

Speechly CLI reference