POST
/
v1
/
http
/
execute-request
{
  "jobs": [
    {}
  ],
  "data_object_id": "<string>",
  "400 Bad Request": {},
  "500 Internal Server Error": {}
}

​
Execute HTTP Request

This endpoint executes an HTTP request to an external API, creates data objects for both the request configuration and response data, establishes data lineage between them, and starts background jobs to analyze and prepare the data.

​
Request Body

​
project_id
string
required

ID of the project to associate with the request

​
datasource_id
string
required

ID of the datasource created for this HTTP connection

​
connect_spec
object
required

Connection specification object containing configuration details

​
url
string
required

The URL of the API endpoint

​
method
string
required

The HTTP method to use

​
headers
object

HTTP headers to include with the request

​
parameters
object

Query parameters with values and required flags

​
parameterGroups
array

Groups of related parameters

​
authType
string

Authentication type

​
auth
object

Authentication details

​
responsePathExtractor
string

JSON path to extract a specific key from the response to use as the main data source. For example, “value” will extract data from the “value” field of the response.

​
Response

​
jobs
array

Array of job objects created for processing the data

​
data_object_id
string

ID of the data object created for the response data

​
Request Example

{
  "url": "https://api.weather.com/v3/location/search",
  "method": "GET",
  "headers": {},
  "parameters": {
    "query": {
      "value": "London",
      "required": true
    },
    "apiKey": {
      "value": "abcdef123456",
      "required": true
    }
  },
  "parameterGroups": [],
  "authType": "None",
  "auth": {},
  "responsePathExtractor": "value",
  "project_id": "9fd3cedc-d303-4870-9635-c89e60c3f1b0",
  "datasource_id": "f4cfd57d-dcf0-4039-ab31-3e2e7178189e",
  "connect_spec": {
    "name": "Weather API Connection",
    "id": "http-requests",
    "config": {
      "url": "https://api.weather.com/v3/location/search",
      "method": "GET",
      "headers": {},
      "parameters": {
        "query": {
          "value": "London",
          "required": true
        },
        "apiKey": {
          "value": "abcdef123456",
          "required": true
        }
      },
      "parameterGroups": [],
      "authType": "None",
      "auth": {},
      "responsePathExtractor": "value"
    }
  }
}

​
Response Example

{
  "jobs": [
    {
      "id": "job-123456",
      "job_type": "build_dataline_from_connected_resource",
      "status": "queued",
      "project_id": "9fd3cedc-d303-4870-9635-c89e60c3f1b0",
      "source": "datasource",
      "source_id": "f4cfd57d-dcf0-4039-ab31-3e2e7178189e",
      "source_event_type": "http_api",
      "created_at": "2023-09-21T15:35:00Z"
    }
  ],
  "data_object_id": "abcdef-1234-5678-9abc-def123456789"
}

​
Error Responses

​
400 Bad Request
object

Returned when the request is malformed or invalid

​
500 Internal Server Error
object

Returned when there’s a server error processing the request

​
Under the Hood

When you call this endpoint, the system:

  1. Creates an HTTPEndpoint instance from your connect_spec
  2. Loads data from the API using endpoint.load_with_defaults()
  3. If a responsePathExtractor is provided, extracts data from the specified key in the response
  4. Stores the HTTP request configuration as a JSON file
  5. Converts the response data (or extracted data) to a Parquet file
  6. Creates data objects in the database for both files
  7. Establishes data lineage connections between the objects
  8. Creates and queues background jobs to analyze and prepare the data

The resulting data object is ready to be used as a datasource in your project.

​
Important Notes

  • The responsePathExtractor parameter should only be used in Step 1 (Test Connection) and Step 4 (Execute HTTP Request). It is not needed in Steps 2 and 3.
  • Use the responsePathExtractor when your API response contains a specific key that holds the main data you want to work with. For example, if your API returns JSON with a value field containing stringified JSON data, setting responsePathExtractor to “value” will extract and use that data.