Vdocuments - MX - SGL Calling Watson From RPG Scott Klement Watson Docs Use Curl Docs Are
Vdocuments - MX - SGL Calling Watson From RPG Scott Klement Watson Docs Use Curl Docs Are
Presented by
Scott Klement
http://www.profoundlogic.com
The Agenda
Agenda for this session:
1. What Is Watson?
2
What is Watson?
https://www.ibm.com/watson/products-services/
6
Inside Your IBM Cloud Account
https://console.bluemix.net/catalog
https://console.bluemix.net/dashboard/apps
8
Create a Service on your IBM Cloud
These are the fields you'll need to provide. (You can usually just take the defaults.)
• Service Name = Unique name for your newly created service. IBM Cloud will
generate a name, but you can change it to whatever you like.
• Region to deploy in = Which IBM Cloud server location. Pick the one closest to
you for best performance.
model_id = en-es
text = Hello
Hola
10
Lets Do it In RPG!
11
• Show screen
• Get data from the screen and build strings
• Use HTTPAPI (or another HTTP tool) to send
and receive
• Put the response string on the screen
12
Watson Docs – Use Curl
Docs are available for Curl, Node.js, Java and Python (but not RPG!)
Curl is the closest to plain HTTP, and easy to adapt to HTTPAPI
13
Curl Is:
• command-line tool for HTTP and many other network protocols
• freely available (open source)
• most commonly found on Linux
• but, can run on IBM i in PASE (AIX version)
• or on Windows, Mac OS, Linux, FreeBSD, etc.
The Curl Home Page:
https://curl.haxx.se
The Curl Manual (to understand what the options do):
https://curl.haxx.se/docs/manpage.html
You will mainly use curl to test out and understand the Watson services, but
will adapt them to HTTPAPI (or another HTTP tool for RPG) for production
use. For that reason, I recommend running it on your PC.
Or, use the Watson API Explorer (next slide)
14
Watson API Explorer
The API Explorer (links are in API docs) provides a Web-based curl interface
15
About HTTPAPI
16
Adapting Curl to HTTPAPI
Use of @ to get from file Use http_req or http_stmf with SendStmf parameter
17
full code sample can be downloaded from Scott's web site (link at end)
http_setAuth( HTTP_AUTH_BASIC
: 'api key from Watson API credentials'
: 'password from Watson API credentials' );
url = 'https://gateway.watsonplatform.net'
+ '/language-translator/api/v2/translate'
+ '?model_id=' + http_urlEncode(fromLang + '-' + toLang)
+ '&text=' + http_urlEncode(%trimr(fromText));
monitor;
resp = http_string('GET': url);
on-error;
httpcode = http_error();
endmon;
18
Even Better, Use JSON
Instead of the "simple" (plain text) format we've used so far, I recommend JSON
• Much more versatile
• Many of the APIs require it
• Works with POST methods, allowing much larger data to be passed
{
"model_id": "en-es",
"text": "Hello"
}
19
{
"source": "en",
"target": "es",
"text": "Hello"
}
{
"translations": [{
"translation": "Hola"
}],
"word_count": 1,
"character_count": 5
}
20
Generating JSON with YAJL
yajl_genOpen(*off);
yajl_beginObj(); // {
yajl_addChar('source': fromLang); // "source": "en",
yajl_addChar('target': toLang); // "target": "es",
yajl_addChar('text': fromText ); // "text": "Hello"
yajl_endObj(); // }
request = yajl_copyBufStr();
yajl_genClose();
Very quick introduction to the YAJL tools for generating JSON documents:
21
Very quick introduction to the YAJL tools for reading JSON documents:
• yajl_string_load_tree loads the whole JSON document intro memory and returns
the "document node" (pointer to outermost JSON element)
• yajl_object_find given a pointer to an object, finds a subfield in that object
• yajl_array_elem given a pointer to an array, finds an element in that array
• yajl_get_string given a pointer to a string, returns the string value
• yajl_get_number, yajl_is_true, yajl_is_false can be used for other data types
• yajl_tree_free removes the JSON document from memory
22
Media Types (aka MIME types)
Various internet protocols use media types (also called "MIME" types) to identify the data
type of something.
These HTTP headers can be used with the Language Translator to tell Watson we want
to send/receive JSON documents instead of plain text.
• content-type header = the type of the document we're sending
• accept header = the type of the document we want to receive back
23
curl -u "{username}":"{password}"
-X POST
-H "Content-Type: application/json"
-H "Accept: application/json"
-d '{"text":"Hello","source":"en","target":"es"}'
"https://gateway.watsonplatform.net/language-translator/api/v2/translate"
dcl-pi *n;
header varchar(32767);
end-pi;
dcl-c CRLF x'0d25';
end-proc;
25
**free
ctl-opt option(*srcstmt) dftactGrp(*no)
bnddir('HTTPAPI':'YAJL');
BNDDIR is used to bind
your program to the tools
/copy httpapi_h
/copy yajl_h
Copybooks contain the
dcl-f WATSONTR1D workstn indds(dspf); definitions we'll need to
call the HTTPAPI and
YAJL routines
dcl-Ds dspf qualified;
F3Exit ind pos(3);
end-Ds;
fromLang = 'en';
toLang = 'es';
26
Translate with JSON (2 of 6)
Main loop controls the flow of the program, repeating the screen until F3 key is
pressed.
exfmt screen1;
if dspf.F3exit = *on;
leave;
endif;
fromLang = %xlate(UPPER:lower:fromLang);
toLang = %xlate(UPPER:lower:toLang);
toText = translate( fromLang: toLang: %trim(fromText) );
enddo;
*inlr = *on;
return;
dcl-pi *n varchar(1000);
fromLang char(2) const;
tolang char(2) const;
fromText varchar(1000) const;
end-pi;
yajl_genOpen(*off);
yajl_beginObj(); // {
yajl_addChar('source': fromLang); // "source": "en",
yajl_addChar('target': toLang); // "target": "fr",
yajl_addChar('text': fromText ); // "text": "String here"
yajl_endObj(); // }
Put the JSON data into a variable
request = yajl_copyBufStr();
named "request"
yajl_genClose(); 28
Translate with JSON (4 of 6)
http_debug(*on: '/tmp/watson-diagnostic-log.txt');
Translate our job's
http_setOption('local-ccsid': '0');
EBCDIC to UTF-8
http_setOption('network-ccsid': '1208');
(standard for web)
http_xproc( HTTP_POINT_ADDL_HEADER: %paddr(AddAccept));
url = 'https://gateway.watsonplatform.net'
+ '/language-translator/api/v2/translate';
monitor;
response = http_string('POST': url: request: 'application/json');
on-error;
httpcode = http_error();
endmon;
http_string will send the "request"
variable we made, and receive
Watson's output into "response"
We first get the "translations" subfield of the document, which is itself an array
of objects. We'll take the first array element, and get it's "translation" subfield,
and then get a string from that.
30
Translate with JSON (6 of 6)
All that's left is the AddAccept subprocedure (that I showed you several slides
back)
dcl-proc AddAccept;
dcl-pi *n;
header varchar(32767);
end-pi;
dcl-c CRLF x'0d25';
end-proc;
31
Following the same techniques will allow you to call any of Watson's services!
32
Example of Finding Approved Vendors
33
This is what the "AVLIST" (approved vendor list) table (PF) looks like (excerpt)
34
Natural Language Understanding (1 of 5)
This is the main loop that controls the program to analyze the vendor's sites:
exec SQL declare C1 cursor for
select * from AVLIST;
request = createJsonRequest(AVLIST);
monitor;
response = http_string('POST': url: request: 'application/json');
on-error;
response = '';
endmon;
extractKeywords(AVLIST.VNUM: response);
dcl-pi *N varchar(1000);
VEND likeds(AVLIST) const;
end-pi;
yajl_genOpen(*off);
yajl_beginObj(); // {
yajl_addChar('url': %trim(VEND.VURL)); // "url": "http://example.com",
yajl_beginObj('features'); // "features" : {
yajl_beginObj('keywords'); // "keywords": {
yajl_endObj(); // }
yajl_endObj(); // }
yajl_endObj(); //
json = yajl_copyBufStr();
yajl_genClose();
return json;
end-proc;
36
Natural Language Understanding (3 of 5)
The "extractKeywords" subprocedure interprets the JSON document that
Watson sent back, gets the keywords that it found, and writes those keywords to a
second table named AVKEYWORDS.
This AVKEYWORDS table (PF) can then be searched by the user.
dcl-proc extractKeywords;
dcl-pi *N;
vnum packed(7: 0) value;
json varchar(100000) const;
end-pi;
37
39
There is setup involved. You must "train" Watson in how to recognize a given
subject.
Once trained, however, it can recognize the same subject in any picture, even
from different angles.
40
Training the Visual Recognition Tool
Training involves:
• Upload .ZIP files with matching
images.
• Upload .ZIP file(s) with "negative"
matches (to teach Watson what not
to consider a match)
41
This example started with an old, green-screen, RPG program for entering
insurance claims (such as car accidents)
42
Insurance Claim Example (Background Info)
Years later (but still awhile back) this application was converted to a web-based
GUI using Profound Logic's tools. RPG Open Access was used so the RPG
code didn't have to change, though a small amount of code was added to allow
picture uploads.
43
44
What Watson Determines
We wrote a routine named watson_classify for our RPG program. It accepts the
pathname to the photograph (in the IFS) and returns the following data structure:
D classify_t ds qualified
D template
D code 10i 0 inz(1)
D errMsg 500a varying inz('')
D class 256a varying inz('')
D score 9p 7 inz(0)
D obj ds likeds(classify_t)
45
C If UploadInfo = '001'
C Eval done = *Off
// --------------------------------------------------------
// New Code for recognizing image:
C eval obj = watson_classify(imagefile)
C if obj.score > 0.75
C
C select
C when obj.class = 'motorcycleaccident'
C eval cmmotor = 'Y'
C when obj.class = 'brokenwinshield'
C eval cmbrokenw = 'Y'
...etc...
If Watson wasn't 75% sure, we ignored it's classification (leaving it to the user)
46
Visual Recognition RPG (1 of 6)
dcl-proc watson_classify;
dcl-pi *n likeds(classify_t);
imageName varchar(256) const;
end-pi;
dcl-c WATSON_API_KEY 'the api key from IBM Cloud is put here';
47
http_debug(*on: '/tmp/watson-claim15r.txt');
imagePath = '/www/profoundui/htdocs/profoundui/userdata/'
+ 'images/claims/' + %trim(imageName);
yajl_genOpen(*off);
yajl_beginObj(); // {
yajl_beginArray('classifier_ids'); // "classifier_ids": [
yajl_addChar('insuranceclaims_1650517727'); // "id-goes-here"
yajl_endArray(); // ]
yajl_endObj(); // }
params = yajl_copyBufStr();
yajl_genClose();
The input JSON document is very simple – it just tells Watson which "classifier"
to use (i.e. which set of data that we trained Watson with)
48
Visual Recognition RPG (3 of 6)
// Create a multipart/form-data form (like curl -F switch)
// to contain both the image and the parameters
// (this is created in a temporary IFS file)
tempFile = http_tempfile();
form = http_mfd_encoder_open( tempFile: contentType );
http_mfd_encoder_addstmf( form
: 'images_file'
: imagePath
: 'image/jpeg' );
http_mfd_encoder_addvar_s( form: 'parameters': params );
http_mfd_encoder_close(form);
The image is uploaded using a "multi part form", like a web browser would use.
• One part contains the input JSON document (from last slide)
• The other part contains the image.
HTTPAPI's multi-part form data (MFD) tool creates this form in a temporary IFS
file. Although the images in the insurance claims are small, HTTPAPI has the
capacity to handle multiple gigabytes of data, so keeping the form in memory
isn't practical.
49
rc = http_req( 'POST'
: url
: *omit: response
: tempFile: *omit
: %trim(contentType) );
Since the input is an IFS file, but I wanted the output to be returned as a string, I
used the http_req() routine.
There are two parameters for "response data", for string and file, respectively.
Also two for "send data", a string and a file.
One send option and one response option must be set to *omit.
50
Response JSON Document
{ We only upload one
"custom_classes":5, image at a time, so there
"images":[ will never be more than
{ one entry in the images
"classifiers":[ array.
{
"classes":[ Only asked for one
{ classifier (trained
"class":"brokenwinshield", databased) so
"classifiers" will only have
"score":0.976408 one array entry.
}
],
"classifier_id":"insuranceclaims_1650517727", Watson might see more
"name":"insurance-claims" than one class. But the
} "top pick" will be first.
],
"image":"image path name here"
} Conclusion: We want the first
], array entry inside the first array
"images_processed":1 entry of the first array entry!
}
51
52
Visual Recognition RPG (6 of 6)
endif;
yajl_tree_free(docNode);
return result;
end-proc;
53
Conclusion
This presentation barely scratches the surface of what Watson can do!
• Cognitive Computing
• A computer that can "think"!
• Understand human language better than ever before
• Understand human photographs better than ever before
• Can search and understand massive volumes of (unstructured)
documents and pictures and discover trends, patterns, etc.
• Best of all, anyone can use it – from RPG on IBM i!
• It's just a web service (API) call!
54
This Presentation
Thank you!
55