README.md

# Extended HTTP Requests

With Extended HTTP Requests, additional functionalities are made available in
order to perform different types of HTTP/S requests (GET, PUT, POST, etc...)
with more control about the content that is sent (headers and body). The
requests can be performed directly from the ADOxx tool using AdoScript. This
functionality enables to retrieve data from resources on the internet or 
performing API calls to web services with custom content.



## Table of contents
1. [Details](#details)
2. [Use](#use)
3. [Extend](#extend)



## Details
[To Top](#)

This module enables the execution of HTTP/S requests from ADOxx or an ADOxx
based modelling tool using AdoScript. It provides several procedures, which
allow to perform a variety of different options to access resources and
services. The procedures use several parameters to handle things like
credentials, headers or the content body for both requests and their responses.

Have a look at the following
[video/screencast](https://www.youtube.com/watch?v=_axMHumFdzQ) to see it in
action! For more information on the application, please visit
https://bee-up.omilab.org/activities/bee-up/scenario-details/


### Features and functionality

1. Perform synchronous HTTP/S calls to Web resources, services etc. using any
desired HTTP method (such as GET, PUT, POST, DELETE, etc.) through AdoScript.
2. (Almost) complete freedom about the sent request headers and body.
3. Use of basic authentication (username and password) supported.
4. Receive the response body in one of three formats:
    * As a string,
    * as a base64 encoded string (necessary when the response contains NUL
      bytes, e.g. PNG images) or
    * as an array of bytes.
5. Additional utility procedures for dealing with common HTTP tasks.


### Cases

[Work in Progress]


### How it works

This implementation uses a Dynamic Link Library (DLL), which is called by the
provided AdoScript procedures. As such, the actual HTTP/S request is performed
from the DLL and the AdoScript procedures provide a neat interface to the DLL.
The DLL itself uses libcurl to handle the network communication.

The following image shows the interaction when performing an HTTP/S call.  
![Composition of the Extended HTTP Requests.](Composition_v3.png)

The **Custom AdoScript** is using the Extended HTTP Requests. It wants to
access something from the Web, Internet or any kind of accessible network using
the Hypertext Transfer Protocol. To achieve this, it invokes the provided
AdoScript procedures.

The **Provided AdoScript Procedures** represent the procedures made available
to simplify calling the DLL, like using AdoScript maps to handle the HTTP/S
request/response headers.

The **ADOxx / ADOxx Modelling Tool** is the environment where the AdoScripts
are executed and which handles the details of calling the DLL and receiving the
result among other things, like creating and managing models.

The **Custom DLL** opens a connection, sends the request to the desired
endpoint on the Web, Internet or any kind of accessible network and waits for
the response. It provides several functions which are available through the C
type calling convention.


#### Libraries and modules
The project is mainly a wrapper around libcurl to simplify HTTP/S requests. It
provides functions for use from other software (e.g. ADOxx) to make synchronous
HTTP/S requests with a single function call. Furthermore it uses a library
(nlohmann\json) for handling JSON data structures where needed (e.g. HTTP
Headers as simple JSON objects). Additionally, a library (base64) for encoding
and decoding of strings (character data) as Base64 is being used.


### Further details / Acknowledgements

* [libcurl](https://curl.se/libcurl/)
* This product includes software developed by the OpenSSL Project for use in
  the OpenSSL Toolkit (http://www.openssl.org/).
* [nlohmann\json](https://github.com/nlohmann/json)
* [base64 encoder/decoder](https://renenyffenegger.ch/notes/development/Base64/Encoding-and-decoding-base-64-with-cpp/)


### Appeal

If you use the Software then we would kindly ask that you do one or several of
the following:
* provide a notice and a link to the original project page on your webpage
  and/or part of the application it is used in at a fitting location.
* use the provided "powered-by-OMiLAB" image found under "res/imgs" on your
  webpage and/or part of the application it is used in at a fitting location.




## Use
[To Top](#)

The Extended HTTP Requests building block can be integrated in your modelling
tool implementation by downloading the necessary files and integrating them
into your library and modelling tool installation.


### Dependencies on other modules / Components

* libcurl (included in download)
* OpenSSL (included in download)
* Other C++ libraries (included in download)


### License of this code
For the license applying to the source code and the compiled binaries created
in this project see `LICENSE`.


### Other licenses
The license details of used libraries, modules etc. can be found in the 
`licenses` directory.


### Using the DLL directly

The DLL can be used directly from any program that supports executing DLL
functions using C deceleration calling conventions. Documentation on the
available functions can be found in `src\HttpRequestDll.h`.

A compiled version for Windows 32-bit is available in the `httpreq` folder as
`HttpRequestFunctions.dll`, together with the necessary DLLs, like
`libcurl.dll`, curl's certificate bundle file `curl-ca-bundle.crt` needed for
HTTPS (both from version 7.77.0-win32) or the libraries from Mingw-w64. For
more information about libcurl visit https://curl.se/libcurl/. For more
information about Mingw-w64 visit http://mingw-w64.org.

> Additional details beyond the here provided descriptions are documented as
> comments in the files located in the `src` folder.


### Setting up in ADOxx (integrated)

Additionally, `ASC_HttpRequestDll.asc` is provided for the use with ADOxx. It
contains an AdoScript with global variables and procedures to perform HTTP/S
calls through the `HttpRequestFunctions.dll`. To use it the intended way
perform the following steps:
1. Copy the `httpreq` folder into the ADOxx installation folder.

2. Add the file `ASC_HttpRequestDll.asc` (located in the `httpreq` folder) to
    the database of the ADOxx library.

3. (recommended) Add the files from the `em` folder to the database of the
    ADOxx library. They contain a list of all used identifiers for global
    variables, functions and procedures.

4. Add the following line in the library's `External coupling` in
    `ON_EVENT "AppInitialized"` at an appropriate place.
    ```AdoScript
    EXECUTE file:("db:\\ASC_HttpRequestDll.asc")
    ```
    This loads the necessary procedures on start-up of the tool. It is
    recommended to load the procedures before any other modules that depend
    on it.


### Using in ADOxx

The script file provides several procedures to execute HTTP/S requests
considering several aspects (detailed further below):
* Authentication.
* Encoding of the response (needed to deal with NUL characters).
* Encoding of the request (needed to deal with NUL characters).

> Additional details beyond the here provided descriptions are documented as
> comments in the AdoScript files located in the `httpreq` folder.

Furthermore, "Utility procedures" are provided to simplify some common tasks
when performing HTTP/S requests:
* `HTTP_URL_ENCODE` - escape unsafe characters in a text using URL encoding.
* `HTTP_URL_ENCODE_QUERY` - transform a map to a string following the structure
    of a URL query, e.g. for sending `application/x-www-form-urlencoded` data.

Additional aspects can be configured through global variables:
* `global_str_dll_dllfolder` – specifies the folder where the necessary DLL
    files are located. Default uses the `httpreq` folder in the ADOxx 
    installation directory.
* `global_str_dll_httprequest` – specifies the name of the DLL file. Default is
    `HttpRequestFunctions.dll`
* `global_val_skipVerification` - if set to 1 then the verification of
    certificates for HTTPS calls will be skipped. Default is 0.

Depending on the used procedure the following parameters are available:
* `str_url` - The main parameter should specify the URL where the request
    should be sent. In case of GET requests the "HTTP parameters" should be
    part of the URL query.
* `str_method` - Which HTTP method should be used. Common are GET, PUT, POST,
    DELETE, but also HEAD, OPTIONS, TRACE and others are possible. Just
    remember that the standardized HTTP methods should use upper case.
* `str_username` - The user name to use with basic authentication.
* `str_password` - The password to use with basic authentication.
* `map_reqheaders` - A map containing the headers to be sent with this request.
    Some seem to always be present automatically when needed (e.g. "Accept",
    "Content-Length"), but others like "Content-Type" or "Authorization"
    should be specified through this map. In any case both the keys and the
    values of the map should be strings, or if you don't want to specify any
    additional headers just use (map()) for this parameter.
* `str_reqbody` - The body that should be sent with the request. In case of
    "PUT" and "POST" requests it should contain the desired parameters, which
    can be simple in some cases (e.g. a string containing valid JSON code) or
    more complicated in others (e.g. a application/x-www-form-urlencoded 
    request has to url-encode the parameters). It is up to the procedure
    caller to produce the proper body content.
* `val_respcode` - a reference that will contain the HTTP status code answered
    by the server or possibly 0 if something went wrong before the request
    could be sent / the response be received.
* `map_respheaders` - a reference that will contain an AdoScript map with the
    response headers. Both the keys and the values will be of type string.
* `str_respbody` / `arr_respbody` - a reference that will contain the response
    of the server or the error message (format depends on used procedure).

The following global procedures are available from `ASC_HttpRequestDll.asc`:
```AdoScript
# Sends an HTTP/S Request to the specified URL. The returned body of the
# response is provided as a string terminated with a NUL byte (\0).
HTTP_SEND_REQUEST (str_url)
    str_method:string
    map_reqheaders:map
    str_reqbody:string
    val_respcode:reference
    map_respheaders:reference
    str_respbody:reference

# Sends an HTTP/S Request to the specified URL with authentication information.
# The returned body of the response is provided as a string terminated with
# a NUL byte (\0).
HTTP_SEND_AUTH_REQUEST (str_url)
    str_method:string
    str_username:string
    str_password:string
    map_reqheaders:map
    str_reqbody:string
    val_respcode:reference
    map_respheaders:reference
    str_respbody:reference

# Sends an HTTP/S Request to the specified URL. The returned body of the
# response is provided as a base64 encoded string. This is relevant for content
# that can contain a NUL byte (\0) in the middle, since the NUL byte is used to
# indicate the end of a string. For example when retrieving a PNG image.
HTTP_SEND_REQUEST_BASE (str_url)
    str_method:string
    map_reqheaders:map
    str_reqbody:string
    val_respcode:reference
    map_respheaders:reference
    str_respbody:reference

# Sends an HTTP/S Request to the specified URL with authentication information.
# The returned body of the response is provided as a base64 encoded string.
# This is relevant for content that can contain a NUL byte (\0) in the middle,
# since the NUL byte is used to indicate the end of a string. For example when
# retrieving a PNG image.
HTTP_SEND_AUTH_REQUEST_BASE (str_url)
    str_method:string
    str_username:string
    str_password:string
    map_reqheaders:map
    str_reqbody:string
    val_respcode:reference
    map_respheaders:reference
    str_respbody:reference

# Sends an HTTP/S Request to the specified URL. The returned body of the
# response is provided as an array containing the byte values. This is relevant
# for content that can contain a NUL byte (\0) in the middle, since the NUL
# byte is used to indicate the end of a string. For example when retrieving a
# PNG image.
HTTP_SEND_REQUEST_BYTES (str_url)
    str_method:string
    map_reqheaders:map
    str_reqbody:string
    val_respcode:reference
    map_respheaders:reference
    arr_respbody:reference

# Sends an HTTP/S Request to the specified URL with authentication information.
# The returned body of the response is provided as an array containing the byte
# values. This is relevant for content that can contain a NUL byte (\0) in the
# middle, since the NUL byte is used to indicate the end of a string. For
# example when retrieving a PNG image.
HTTP_SEND_AUTH_REQUEST_BYTES (str_url)
    str_method:string
    str_username:string
    str_password:string
    map_reqheaders:map
    str_reqbody:string
    val_respcode:reference
    map_respheaders:reference
    arr_respbody:reference

###############################################################################
# The following procedures are a special version of the previous procedures
# which will decode the str_reqbody using base64 before sending it to the
# specified URL. This is again relevant when sending data that can contain a
# NUL byte (\0) in the middle. Otherwise they behave the same.
###############################################################################

HTTP_SEND_REQUEST_INBASE (str_url)
    str_method:string
    map_reqheaders:map
    str_reqbody:string
    val_respcode:reference
    map_respheaders:reference
    str_respbody:reference

HTTP_SEND_AUTH_REQUEST_INBASE (str_url)
    str_method:string
    str_username:string
    str_password:string
    map_reqheaders:map
    str_reqbody:string
    val_respcode:reference
    map_respheaders:reference
    str_respbody:reference

HTTP_SEND_REQUEST_BASE_INBASE (str_url)
    str_method:string
    map_reqheaders:map
    str_reqbody:string
    val_respcode:reference
    map_respheaders:reference
    str_respbody:reference

HTTP_SEND_AUTH_REQUEST_BASE_INBASE (str_url)
    str_method:string
    str_username:string
    str_password:string
    map_reqheaders:map
    str_reqbody:string
    val_respcode:reference
    map_respheaders:reference
    str_respbody:reference

HTTP_SEND_REQUEST_BYTES_INBASE (str_url)
    str_method:string
    map_reqheaders:map
    str_reqbody:string
    val_respcode:reference
    map_respheaders:reference
    arr_respbody:reference

HTTP_SEND_AUTH_REQUEST_BYTES_INBASE (str_url)
    str_method:string
    str_username:string
    str_password:string
    map_reqheaders:map
    str_reqbody:string
    val_respcode:reference
    map_respheaders:reference
    arr_respbody:reference


###############################################################################
# Following the utility procedures.
###############################################################################


HTTP_URL_ENCODE (str_content)
    str_allowedset:string
    str_encoded:reference
# str_content - The content that should be encoded using URL encoding style.
#   Consider providing the main parameter with the toutf8(...) function.
# str_allowedset - Specifies which of the four available sets containing the
#   "allowed characters" to use. Should be "JS", "Inter", "Lax" or "VeryLax".
# str_encoded - a reference that will contain the encoded content.

HTTP_URL_ENCODE_QUERY (map_content)
      str_allowedset:string
      str_encoded:reference
# map_content - a map containing keys and their values to transform.
# str_allowedset - Specifies which of the four available sets containing the
#   "allowed characters" to use. Should be "JS", "Inter", "Lax" or "VeryLax".
# str_encoded - a reference that will contain the encoded content.
```

Additional details on the available AdoScript procedures are documented in the
`ASC_HttpRequestDll.asc` file.


### Backwards compatibility

The `HttpRequestFunctions.dll` also provides functions that have the same
signature and functionality as the old `HttpRequestDll.dll`. Therefore it can
be used in place of the old version, however the necessary dependencies
(`libcurl`, `curl-ca-bundle.crt`) must be placed in the same folder.


### Known issues

* During initial testing under Linux (e.g. Ubuntu) it sometimes lead to crashes
    of the ADOxx tool. So far the details on how to reproduce this are not
    known (seems to happen at random). Might only be a problem with the testing
    environment (virtual machine, bad tests, used wine version etc.).
    If you have any additional details then please provide them to us.


### Other notes
* When using the DLLs directly from ADOxx it is necessary to change the
    "Current Working Directory" to the directory where they are located. This
    is not necessary when using the procedures provided in
    `ASC_HttpRequestDll.asc`.
* When creating a modelling tool installation package make sure it also
    contains the `httpreq` folder and is distributed with the installer.



## Extend
[To Top](#)

The core functionality is implemented using C++, which calls other libraries
(e.g. libcurl) and is built into a DLL. This DLL is then called from AdoScript
procedures.

It is recommended to have knowledge about:
* The Hypertext Transfer Protocol.
* C++ and building C++ projects.
* libcurl and its interfaces.
* ADOxx and AdoScript, especially maps, calling DLLs and translating between
    ADOxx data types.


### Setup of development environment

The following setup has been used to develop the project and has worked so far:
1. [Mingw-w64](http://mingw-w64.org): the compiler used for building the C++
    project. It should be installed for i686 architecture (32-bit), using posix
    threads and dwarf exception handling. The version used for building was
    8.1.0.
2. [Visual Studio Code](https://code.visualstudio.com/): Code editor for which
    several extensions are available for working with the relevant files.
3. [C/C++ for Visual Studio Code](https://marketplace.visualstudio.com/items?itemName=ms-vscode.cpptools):
    Extension for Visual Studio Code to support writing and building C++
    projects.
4. [VSC AdoScript Extension](https://marketplace.visualstudio.com/items?itemName=ADOxxorg.adoxx-adoscript):
    An extension for Visual Studio Code to edit AdoScript and other ADOxx
    formats.


### Adapt the source code
The source code is written in C++ and is provided in the `src` folder. It
consists of three parts:
* HttpRequest - contains structures and classes to perform HTTP/S requests from
    mainly C++. The core in this case is the `LibcurlWrapper` class.
* HttpRequestDll - contains the functions for performing HTTP/S requests made
    available through the DLL using C declaration (cdecl) calling conventions.
    Also contains some other support functions provided by the DLL.
* StringHelper - contains some functions that work on/with C++ strings used by
    the other parts.

The source code for the AdoScript procedures is directly available in the
`ASC_HttpRequestDll.asc` file in the `httpreq` folder. It's main concern is
simplifying the use of the DLL through global procedures.

The files of the used libraries can be found in the `include` directory.


### Build the project
The project has been built for 32-bit using GCC through mingw-w64. So far the
project is not complicated enough to warrant a MAKE file or similar.
Used mingw-w64 version: i686-8.1.0-posix-dwarf-rt_v6-rev0.
Used command (in relation to the project directory):  
`g++.exe -std=c++17 -O3 .\src\*.cpp .\include\*.cpp -o .\httpreq\HttpRequestFunctions.dll -I .\include -L .\lib -l curl -mdll`


### How to contribute

There are different ways to contribute to the project:
* Spread the word and tell the others
* Report issues and problems
* Extend the functionality and fix existing issues

The easiest way to achieve the latter parts is by installing [git](https://git-scm.com/)
and using the [repository of the project](https://code.omilab.org/resources/adoxx-modules/extended-http-requests).

When deciding to implement a new feature or fix a specific bug then create an
"Issue" in the GitLab project repository first. This can be used for discussion
with other community members and help to keep track of work items. Also avoid
performing changes directly to the master branch if you are not the owner or
a maintainer of the project. Instead create a new branch that is descriptive of
the work to be done. Once the desired changes are implemented the result can be
committed to the GitLab repository and a merge request performed.