Welcome to the TM Forum Open API Conformance Testing and Validation Instructions Page!
The following guidelines will take you through the various steps needed to run self-testing for your API to achieve conformance certification.
Self-Certification is based on the TM Forum Open API conformance profiles, it is the process by which an implementation is confirmed to support the standard definition of a TM Forum Open API. Currently available conformance profiles can be found through the following links:
TMF661 Trouble Ticket API Conformance Profile R16.5.0
- TMF660 Product Catalog API Conformance Profile R16.5.0
- TMF659 Customer Management API Conformance Profile R16.5.0
- TMF637 Product Inventory Management API Conformance Profile R16.5
- TMF645 Service Qualification API Conformance Profile R16.5
The TM Forum provides a set of scripts useful to self-certify your implementation of TM Forum Open API specifications. You will need to install and run the applicable API CTK. Currently there is a number of CTKs available for conformance testing and the list is steadily increasing. The following is a list of APIs which have currently CTKs available for conformance testing:
Trouble Ticket CTK (API REST Specification TMF661 - R16.5.0)
Product Catalog Management CTK (API REST Specification TMF660 - R16.5.0)
Customer Management CTK (API REST Specification TMF659 - R16.5.0)
- Product Inventory Management CTK (API REST Specification TMF637 - R16.5.0)
- Service Qualification CTK (API REST Specification TMF645 - R16.5.0)
The following instructions describe how to install an Open API CTK
Tools needed for Open API CTK testing and validation:
Newman (needed for CTK test execution), this is the command line version of 'Postman'; installation instructions available at: https://www.getpostman.com/docs/newman_intro
Newman is built on Node.js. To run Newman, make sure you have Node.js installed. Node.js can be downloaded and installed from here on Linux, Windows, and Mac OSX.
Node.js offers a server-side environment, which allows the use of JavaScript language to generate web pages. Basically, it replaces server languages such as PHP, Java EE, etc. Once Node.js is installed, Newman is just a command away:
npm install -g newmanThis installs Newman from npm globally on your system allowing you to run it from anywhere.
npm makes it easy for JavaScript developers to share and reuse code, and it makes it easy to update the code that you're sharing. (You can download npm from here).
Once Node.js and Newman are installed follow the next steps:
1 Import CTK from TM Forum
- Download and unzip postman collection file for selected API from the corresponding page (see CTK links provided above)
- Store downloaded file into the folder in your system where Newman runs
2 Configure environment
- Download environment file (TMFENV) from the following link: TMFENV File
- Modify environment file to refer to the endpoint where the implementation to be certified is running (see step 3 below)
- Store modified file into the folder in your system where Newman runs
3 You need to create your testing environment. The environment is a text file without extension written in JSON. It has the following format:
You can use this as a template. Create a file with this format, replace the values for the name, the host and your API addresses and put it in the same folder as the postman collections.
4 Now create the right script or batch file for your platform (either Windows or Linux/Mac), put it in the same folder and run script execution command. For example, you can use a batch file (runtest.bat) which would look like the following example:
set http_proxy=http://proxyinternet.tesa:8080
newman -c DSTestedAPITestCollection.postman_collection -e TMFENV -H DSTestedAPICTKResult.html -o DSTestedAPICTKResult.json
pause >nul
5 A folder name results will be created and filled with the output of the tests. You should see something like the following:
If they are no failures, then you have passed the CTK and your API is conformant with all the Mandatory features.
The results of the CTK are in the DSTestedAPICTKResult.html and DSTestedAPICTKResult.json files and should look like the following (example with Customer Management API):
All the information related to the execution of the CTK will be contained in the DSTestedAPICTKResult.json file.
The following is a complete example for the Trouble Ticket API:
Newman test report
Collection: TMForum - TroubleTicket API CTK
Iterations: 1
Description:
Requests:
Description:
Mean time per request: 6123 ms
Total tests that passed: 2
Total tests that failed: 0
Individual tests:
| Test name | Pass count | Fail count |
|---|---|---|
| Content-Type is present application/json | 1 | 0 |
| Status code is 404 | 1 | 0 |
Description: retrieve a product offering
Mean time per request: 3675 ms
Total tests that passed: 2
Total tests that failed: 0
Individual tests:
| Test name | Pass count | Fail count |
|---|---|---|
| Content-Type is present application/json | 1 | 0 |
| Status code is an error | 1 | 0 |
Description: retrieve a product offering
Mean time per request: 4320 ms
Total tests that passed: 2
Total tests that failed: 0
Individual tests:
| Test name | Pass count | Fail count |
|---|---|---|
| Content-Type is present text/plain | 1 | 0 |
| Status code is an error | 1 | 0 |
Description: create a trouble ticket
Mean time per request: 3863 ms
Total tests that passed: 5
Total tests that failed: 0
Individual tests:
| Test name | Pass count | Fail count |
|---|---|---|
| Status code is 201 | 1 | 0 |
| Content-Type is present application/json | 1 | 0 |
| Response contains ID | 1 | 0 |
| Response contains all required fields | 1 | 0 |
| POST Body Response equals Request Body | 1 | 0 |
Description: retrieve /troubleTickets
Mean time per request: 7652 ms
Total tests that passed: 5
Total tests that failed: 0
Individual tests:
| Test name | Pass count | Fail count |
|---|---|---|
| Content-Type is present application/json | 1 | 0 |
| Status code is 200 | 1 | 0 |
| Response contains all required fields | 1 | 0 |
| Response contains created resource | 1 | 0 |
| Element contained equals previous creation | 1 | 0 |
Description: retrieve a troubleTicket
Mean time per request: 1300 ms
Total tests that passed: 6
Total tests that failed: 0
Individual tests:
| Test name | Pass count | Fail count |
|---|---|---|
| Content-Type is present application/json | 1 | 0 |
| Status code is 200 | 1 | 0 |
| Response contains ID | 1 | 0 |
| Response contains all required fields | 1 | 0 |
| ID is correct | 1 | 0 |
| Body Response equals previous creation | 1 | 0 |
Description: create a trouble ticket
Mean time per request: 1315 ms
Total tests that passed: 5
Total tests that failed: 0
Individual tests:
| Test name | Pass count | Fail count |
|---|---|---|
| Status code is 201 | 1 | 0 |
| Content-Type is present application/json | 1 | 0 |
| Response contains ID | 1 | 0 |
| Response contains all required fields | 1 | 0 |
| POST Body Response equals Request Body | 1 | 0 |
Description: retrieve /troubleTickets
Mean time per request: 5085 ms
Total tests that passed: 5
Total tests that failed: 0
Individual tests:
| Test name | Pass count | Fail count |
|---|---|---|
| Content-Type is present application/json | 1 | 0 |
| Status code is 200 | 1 | 0 |
| Response contains all required fields | 1 | 0 |
| Response contains created resource | 1 | 0 |
| Element contained equals previous creation | 1 | 0 |
Description: retrieve a troubleTicket
Mean time per request: 4147 ms
Total tests that passed: 6
Total tests that failed: 0
Individual tests:
| Test name | Pass count | Fail count |
|---|---|---|
| Content-Type is present application/json | 1 | 0 |
| Status code is 200 | 1 | 0 |
| Response contains ID | 1 | 0 |
| Response contains all required fields | 1 | 0 |
| ID is correct | 1 | 0 |
| Body Response equals previous creation | 1 | 0 |
Description: retrieve troubleTickets
Mean time per request: 4481 ms
Total tests that passed: 7
Total tests that failed: 0
Individual tests:
| Test name | Pass count | Fail count |
|---|---|---|
| Content-Type is present application/json | 1 | 0 |
| Status code is 200 | 1 | 0 |
| Response contains all required fields | 1 | 0 |
| Response contains created resource 1 | 1 | 0 |
| Element contained equals previous creation 1 | 1 | 0 |
| Response contains created resource 2 | 1 | 0 |
| Element contained equals previous creation 2 | 1 | 0 |
Description: retrieve tickets with high severity
Mean time per request: 2135 ms
Total tests that passed: 6
Total tests that failed: 0
Individual tests:
| Test name | Pass count | Fail count |
|---|---|---|
| Content-Type is present application/json | 1 | 0 |
| Status code is 200 | 1 | 0 |
| Response contains all required fields | 1 | 0 |
| Response does not contain created resource 2 | 1 | 0 |
| Response contains created resource 1 | 1 | 0 |
| Element contained equals previous creation 1 | 1 | 0 |
Description: retrieve tickets with connectivity problems
Mean time per request: 1560 ms
Total tests that passed: 6
Total tests that failed: 0
Individual tests:
| Test name | Pass count | Fail count |
|---|---|---|
| Content-Type is present application/json | 1 | 0 |
| Status code is 200 | 1 | 0 |
| Response contains all required fields | 1 | 0 |
| Response does not contain created resource 1 | 1 | 0 |
| Response contains created resource 2 | 1 | 0 |
| Element contained equals previous creation 2 | 1 | 0 |
Description: retrieve the product offering that are active
Mean time per request: 887 ms
Total tests that passed: 3
Total tests that failed: 0
Individual tests:
| Test name | Pass count | Fail count |
|---|---|---|
| Content-Type is present application/json | 1 | 0 |
| Status code is 200 | 1 | 0 |
| Element contained equals previous creation | 1 | 0 |
Description: retrieve the product offering that are active
Mean time per request: 620 ms
Total tests that passed: 3
Total tests that failed: 0
Individual tests:
| Test name | Pass count | Fail count |
|---|---|---|
| Content-Type is present application/json | 1 | 0 |
| Status code is 200 | 1 | 0 |
| Element contained equals previous creation | 1 | 0 |
Description: retrieve the product offering that are active
Mean time per request: 598 ms
Total tests that passed: 5
Total tests that failed: 0
Individual tests:
| Test name | Pass count | Fail count |
|---|---|---|
| Content-Type is present application/json | 1 | 0 |
| Status code is 200 | 1 | 0 |
| Response contains created resource 1 | 1 | 0 |
| Element contained equals previous creation | 1 | 0 |
| Response does not contain created resource 2 | 1 | 0 |