Skip to content

Commit

Permalink
[4.3] - kazoo_speech: ASR Billing Features and kazoo_asr Refactor (#6185
Browse files Browse the repository at this point in the history
)

* kazoo_speech: ASR Billing Features and kazoo_asr Refactor (#6055)

This PR enhances the kazoo_speech application by augmenting the
`kazoo_asr` abstraction as well as introduce new billing features.

- Add `transcribe` field to the mailbox schema
- Add and export callbacks to`gen_asr_provider` for content-types
- Enhance the `kazoo_asr` behavior to handle default and accepted
  content types on behalf of the specific providers
- Create an abstraction for ASR requests
- Create account ledger entries on successful transcription as well as
  the impact the reseller ledgers.
- Create `asr_flat_rate` module to handle flat rate billing of ASR
  requests.
- Create quantifiers in `kz_services` for asr transcriptions.

```
"asr": {
       "google": {
           "rate": 1,
           "name": "Google ASR"
       },
       "ispeech": {
           "rate": 1,
           "name": "ispeech ASR"
       }
   },
   "plan": {
   ....
       "voicemails": {
           "mailbox": {
               "name": "Voicemail Box",
               "rate": 1.99,
               "cascade": true
           },
           "transcription": {
               "cascade": true,
               "rate": 1,
               "name": "VMBox Transcription MRC"
           }
       },
....
```

The only configuration change in this PR is adding the `transcribe`
flag to the JSON schema.

Two new callbacks have been added to `gen_asr_provider` to help
facilitate a more generic approach to handling the preferred
content-types for a provider as well as act as a gatekeeper and help
identify if conversion is required or even currently supported for the
submitted media.

- preferred_content_type/0
- accepted_content_types/0

This callback is designed to return the ASR provider's preferred
content-type for requests.

This callback is designed to return the accepted content-types and
assist in determining if the media payload will require conversion.

The content-type callbacks have been added to `kazoo_asr` as well any
`kapps_config` calls from the current providers in favor of keeping
with the abstraction.

Originally the ASR logic was hardcoded into the voicemail notify and
save logic in `kvm_util`.  I've removed that and replaced it with an
asr_request.

The `asr_request` module is designed to be a generic request type to
handle creating and servicing ASR requests which are proxied with
`kazoo_asr` to the configured provider..  Additionally it also handles
the billing and services logic and introduces a `asr_flat_rate` module
that can consult an ASR service plan item for rates.

The long term vision is to create a primitive that can in the future
could be augmented to handle multiple types of ASR requests apart from
file conversion (i.e. streaming from an active call).

I've added some placeholder fields in `#asr_req` record should metered
billing be desired.

`asr_flat_rate` is the only and default `billing_method` for an
`asr_request`.

It is patterned after jonny5 and aims to perform the following actions
on an `asr_req`:
- **authorize**
  *account and reseller have funds*
- **debit**
  *ledger entries created for account and reseller if configured*

Some ASR primitives have been gently introduced into `kz_services`: -
`asr` getter methods have been added to `kzd_service_plans` and
`kz_service_plans` - default quantifiers for transcribe enabled
maiboxes in `services.hrl` - `kz_services_asr` module for interacting
with the transcription usage ledgers

* fix merge conflict

* resolve additional merge conflict issues

* swagger and schema doc issues

* more swagger fixups

* appease edoc

* formatting issue

* run make fmt
  • Loading branch information
mk1s authored and k-anderson committed Dec 9, 2019
1 parent 61835ee commit 37c3e4b
Show file tree
Hide file tree
Showing 27 changed files with 1,329 additions and 109 deletions.
1 change: 1 addition & 0 deletions applications/crossbar/doc/ref/vmboxes.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ Key | Description | Type | Default | Required | Support Level
`skip_greeting` | Determines if the greeting should be skipped | `boolean()` | `false` | `false` | `supported`
`skip_instructions` | Determines if the instructions after the greeting and prior to composing a message should be played | `boolean()` | `false` | `false` | `supported`
`timezone` | The default timezone | `string(5..32)` | | `false` | `supported`
`transcribe` | Transcribe voicemail using ASR engine | `boolean()` | `false` | `false` | `supported`

### notify.callback

Expand Down
1 change: 1 addition & 0 deletions applications/crossbar/doc/voicemail.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ Key | Description | Type | Default | Required | Support Level
`skip_greeting` | Determines if the greeting should be skipped | `boolean()` | `false` | `false` | `supported`
`skip_instructions` | Determines if the instructions after the greeting and prior to composing a message should be played | `boolean()` | `false` | `false` | `supported`
`timezone` | The default timezone | `string(5..32)` | | `false` | `supported`
`transcribe` | Transcribe voicemail using ASR engine | `boolean()` | `false` | `false` | `supported`

### notify.callback

Expand Down
34 changes: 34 additions & 0 deletions applications/crossbar/priv/api/swagger.json
Original file line number Diff line number Diff line change
Expand Up @@ -298,6 +298,10 @@
{
"friendly_name": "Support",
"name": "support"
},
{
"friendly_name": "ASR Transcriptions",
"name": "asr-transcriptions"
}
],
"description": "ledgers registered_ledgers",
Expand Down Expand Up @@ -32108,6 +32112,10 @@
{
"friendly_name": "Support",
"name": "support"
},
{
"friendly_name": "ASR Transcriptions",
"name": "asr-transcriptions"
}
],
"description": "ledgers registered_ledgers",
Expand Down Expand Up @@ -34224,10 +34232,26 @@
"description": "API key for Google Cloud Speech",
"type": "string"
},
"asr_enable_automatic_punctuation": {
"default": true,
"description": "detect and insert punctuation in transcription results",
"type": "boolean"
},
"asr_enable_word_time_offsets": {
"description": "top result includes a list of words and the start and end time offsets",
"type": "boolean"
},
"asr_model": {
"default": "phone_call",
"description": "model to select for the given request",
"enum": [
"command_and_search",
"phone_call",
"video",
"default"
],
"type": "string"
},
"asr_profanity_filter": {
"description": "server will attempt to filter out profanities, replacing all but the initial character in each filtered word with asterisk",
"type": "boolean"
Expand All @@ -34237,6 +34261,11 @@
"description": "Google Cloud Speech API url",
"type": "string"
},
"asr_use_enhanced": {
"default": true,
"description": "use an enhanced model for speech recognition",
"type": "boolean"
},
"tts_api_key": {
"default": "",
"description": "speech google tts_api_key",
Expand Down Expand Up @@ -35958,6 +35987,11 @@
"maxLength": 32,
"minLength": 5,
"type": "string"
},
"transcribe": {
"default": false,
"description": "Transcribe voicemail using ASR engine",
"type": "boolean"
}
},
"required": [
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,10 @@
{
"friendly_name": "Support",
"name": "support"
},
{
"friendly_name": "ASR Transcriptions",
"name": "asr-transcriptions"
}
],
"description": "ledgers registered_ledgers",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,10 @@
{
"friendly_name": "Support",
"name": "support"
},
{
"friendly_name": "ASR Transcriptions",
"name": "asr-transcriptions"
}
],
"description": "ledgers registered_ledgers",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,26 @@
"description": "API key for Google Cloud Speech",
"type": "string"
},
"asr_enable_automatic_punctuation": {
"default": true,
"description": "detect and insert punctuation in transcription results",
"type": "boolean"
},
"asr_enable_word_time_offsets": {
"description": "top result includes a list of words and the start and end time offsets",
"type": "boolean"
},
"asr_model": {
"default": "phone_call",
"description": "model to select for the given request",
"enum": [
"command_and_search",
"phone_call",
"video",
"default"
],
"type": "string"
},
"asr_profanity_filter": {
"description": "server will attempt to filter out profanities, replacing all but the initial character in each filtered word with asterisk",
"type": "boolean"
Expand All @@ -21,6 +37,11 @@
"description": "Google Cloud Speech API url",
"type": "string"
},
"asr_use_enhanced": {
"default": true,
"description": "use an enhanced model for speech recognition",
"type": "boolean"
},
"tts_api_key": {
"default": "",
"description": "speech google tts_api_key",
Expand Down
6 changes: 6 additions & 0 deletions applications/crossbar/priv/couchdb/schemas/vmboxes.json
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,12 @@
"minLength": 5,
"support_level": "supported",
"type": "string"
},
"transcribe": {
"default": false,
"description": "Transcribe voicemail using ASR engine",
"support_level": "supported",
"type": "boolean"
}
},
"required": [
Expand Down
5 changes: 5 additions & 0 deletions core/kazoo_documents/src/kzd_box_message.erl
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@

,change_message_name/2, change_to_sip_field/3

,length/1
,media_id/1, set_media_id/2, update_media_id/2
,metadata/1, metadata/2, set_metadata/2
,source_id/1, set_source_id/2
Expand Down Expand Up @@ -263,6 +264,10 @@ message_history(JObj) ->
add_message_history(History, JObj) ->
kz_json:set_value(?KEY_HISTORY, message_history(JObj) ++ [History], JObj).

-spec length(doc()) -> integer().
length(JObj) ->
kz_json:get_value(?KEY_META_LENGTH, JObj).

-spec message_name(doc()) -> kz_term:api_binary().
message_name(JObj) ->
message_name(JObj, 'undefined').
Expand Down
16 changes: 16 additions & 0 deletions core/kazoo_documents/src/kzd_service_plan.erl
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,9 @@
,ratedeck_name/2
,set_ratedeck_name/2
]).
-export([asr/1
,asr/2
]).
-export([applications/1
,applications/2
,set_applications/2
Expand Down Expand Up @@ -279,6 +282,19 @@ applications(JObj, Default) ->
set_applications(JObj, Applications) ->
kz_json:set_value(?APPLICATIONS, Applications, JObj).

%%------------------------------------------------------------------------------
%% @doc
%% @end
%%------------------------------------------------------------------------------
-spec asr(doc()) -> kz_json:object().
asr(JObj) ->
asr(JObj, kz_json:new()).

-spec asr(doc(), Default) -> Default | kz_json:object().
asr(JObj, Default) ->
kz_json:get_ne_json_value(<<"asr">>, JObj, Default).


%%------------------------------------------------------------------------------
%% @doc
%% @end
Expand Down
13 changes: 13 additions & 0 deletions core/kazoo_documents/src/kzd_vmboxes.erl.src
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
-export([skip_greeting/1, skip_greeting/2, set_skip_greeting/2]).
-export([skip_instructions/1, skip_instructions/2, set_skip_instructions/2]).
-export([timezone/1, timezone/2, set_timezone/2]).
-export([transcribe/1, transcribe/2, set_transcribe/2]).


-include("kz_documents.hrl").
Expand Down Expand Up @@ -304,3 +305,15 @@ timezone(Doc, Default) ->
-spec set_timezone(doc(), kz_term:ne_binary()) -> doc().
set_timezone(Doc, Timezone) ->
kz_json:set_value([<<"timezone">>], Timezone, Doc).

-spec transcribe(doc()) -> boolean().
transcribe(Doc) ->
transcribe(Doc, false).

-spec transcribe(doc(), Default) -> boolean() | Default.
transcribe(Doc, Default) ->
kz_json:get_boolean_value([<<"transcribe">>], Doc, Default).

-spec set_transcribe(doc(), boolean()) -> doc().
set_transcribe(Doc, Transcribe) ->
kz_json:set_value([<<"transcribe">>], Transcribe, Doc).
3 changes: 3 additions & 0 deletions core/kazoo_services/src/kazoo_services_maintenance.erl
Original file line number Diff line number Diff line change
Expand Up @@ -211,6 +211,9 @@ js_quantify_map() ->
" break;"
" case 'vmbox':"
" emit(['voicemails', 'mailbox'], 1);"
" if (doc.transcribe) {"
" emit(['voicemails', 'transcription'], 1);"
" }"
" break;"
" case 'faxbox':"
" emit(['faxes', 'mailbox'], 1);"
Expand Down
9 changes: 9 additions & 0 deletions core/kazoo_services/src/kz_services_plan.erl
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
-export([ratedeck_id/1]).
-export([ratedeck_name/1]).
-export([applications/1]).
-export([asr/1]).
-export([limits/1]).
-export([jobj/1
,set_jobj/2
Expand Down Expand Up @@ -169,6 +170,14 @@ ratedeck_name(Plan) ->
applications(Plan) ->
kzd_service_plan:applications(jobj(Plan)).

%%------------------------------------------------------------------------------
%% @doc
%% @end
%%------------------------------------------------------------------------------
-spec asr(plan()) -> kz_json:object().
asr(Plan) ->
kzd_service_plan:asr(jobj(Plan)).

%%------------------------------------------------------------------------------
%% @doc
%% @end
Expand Down
17 changes: 15 additions & 2 deletions core/kazoo_services/src/kz_services_quantities.erl
Original file line number Diff line number Diff line change
Expand Up @@ -552,8 +552,21 @@ calculate_vmbox_updates(JObj, Updates) ->
case kz_doc:type(JObj) =:= <<"vmbox">> of
'false' -> Updates;
'true' ->
Key = [<<"voicemails">>, <<"mailbox">>],
[{Key, 1} | Updates]
Keys = [<<"mailbox">>, <<"transcription">>],
Fun = calculate_vmbox_updates_foldl(JObj),
lists:foldl(Fun, Updates, Keys)
end.

-spec calculate_vmbox_updates_foldl(kz_json:object()) -> fun((kz_term:ne_binary(), kz_term:proplist()) -> kz_term:proplist()).
calculate_vmbox_updates_foldl(JObj) ->
fun(<<"transcription">> = Key, Updates) ->
case kz_json:get_boolean_value(<<"transcribe">>, JObj, 'false') of
'true' ->
[{[<<"voicemails">>, Key], 1} | Updates];
_ -> Updates
end;
(Key, Updates) ->
[{[<<"voicemails">>, Key], 1} | Updates]
end.

-spec calculate_faxbox_updates(kz_json:object(), kz_term:proplist()) -> kz_term:proplist().
Expand Down
61 changes: 61 additions & 0 deletions core/kazoo_services/src/modules/kz_services_asr.erl
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
%%%-----------------------------------------------------------------------------
%%% @copyright (C) 2012-2019, 2600Hz
%%% @doc This Source Code Form is subject to the terms of the Mozilla Public
%%% License, v. 2.0. If a copy of the MPL was not distributed with this
%%% file, You can obtain one at https://mozilla.org/MPL/2.0/.
%%%
%%% @end
%%%-----------------------------------------------------------------------------
-module(kz_services_asr).

-export([fetch/1
,flat_rate/1, flat_rate/2
]).

-include("services.hrl").

-define(DEFAULT_FLAT_RATE, 0).

%%------------------------------------------------------------------------------
%% @doc
%% @end
%%------------------------------------------------------------------------------
-spec fetch(kz_services:services() | kz_term:ne_binary()) -> kz_json:object().
fetch(?NE_BINARY=AccountId) ->
FetchOptions = ['hydrate_plans'],
fetch(kz_services:fetch(AccountId, FetchOptions));
fetch(Services) ->
ASRDict = kz_services_plans:foldl(fun fetch_foldl/3
,dict:new()
,kz_services:plans(Services)
),
kz_json:from_list(dict:to_list(ASRDict)).

%%------------------------------------------------------------------------------
%% @doc
%% @end
%%------------------------------------------------------------------------------
-spec fetch_foldl(kz_term:ne_binary(), kz_services_plans:plans_list(), dict:dict()) -> dict:dict().
fetch_foldl(_BookkeeperHash, [], Providers) ->
Providers;
fetch_foldl(_BookkeeperHash, PlansList, Providers) ->
Plan = kz_services_plans:merge(PlansList),
kz_json:foldl(fun(K, V, A) ->
dict:store(K, V, A)
end
,Providers
,kz_services_plan:asr(Plan)
).

%%------------------------------------------------------------------------------
%% @doc
%% @end
%%------------------------------------------------------------------------------
-spec flat_rate(kz_term:ne_binary()) -> kz_currency:dollars().
flat_rate(AccountId) ->
flat_rate(AccountId, kazoo_asr:default_provider()).

-spec flat_rate(kz_term:ne_binary(), kz_term:ne_binary()) -> kz_currency:dollars().
flat_rate(AccountId, Provider) ->
Items = fetch(AccountId),
kz_json:get_number_value([Provider, <<"rate">>], Items, ?DEFAULT_FLAT_RATE).
4 changes: 3 additions & 1 deletion core/kazoo_services/src/services.hrl
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,9 @@
]
}
,{<<"voicemails">>
,[<<"mailbox">>]
,[<<"mailbox">>
,<<"transcription">>
]
}
,{<<"faxes">>
,[<<"mailbox">>]
Expand Down
Loading

0 comments on commit 37c3e4b

Please sign in to comment.