Article font size: default
← All posts

API integration

Designing a Reliable API Integration

An API integration connects two systems so they can exchange data or trigger actions automatically.

A reliable integration takes more than sending an API request. You must decide which system owns the data, how records are matched, and when synchronization (sync) runs. You must also decide what happens when a request fails, and how the process is secured and monitored.

The main example here is a one-way integration between an ERP, which is a business management system, and an e-shop. The ERP is the source of truth, and product information moves from the ERP to the e-shop.

1. Define the goal of the integration

Start by saying exactly what the integration should do.

Avoid broad goals like these:

  • Connect the ERP with the e-shop.
  • Synchronize the systems.
  • Exchange product information.

They do not explain which data moves, in which direction, or what should happen after it arrives.

A clearer goal looks like this:

  • Sync products from the ERP to the e-shop.
  • Create products in the e-shop when they are created in the ERP.
  • Update prices and stock when they change in the ERP.
  • Disable products in the e-shop when they are no longer active in the ERP.

Also decide what type of integration you need. It can be one-way or two-way. It can also be real-time, scheduled, manually triggered, or event-driven.

Example: The ERP is the source of truth for product codes, names, prices, stock levels, VAT categories, and availability. The e-shop receives these values but does not send product changes back to the ERP.

2. Define the core requirements

Write down the technical and operational requirements before you build anything.

Start with how often sync should run. It could run right after a product changes, every few minutes, every hour, once a day, or only when an administrator starts it.

The right frequency depends on the business need. Stock may need to update every few minutes, while product descriptions may only need to update once a day.

Also estimate:

  • Volume: Number of products, changes per day, and expected API requests.
  • Limits: Available processing power, API rate limits, maximum acceptable delay, and maximum execution time.
  • Growth: How much all of this may increase in the future.

Plan the process so it does not overload either system. Read data in pages, process records in batches, and control how many tasks run at once. Set timeouts for requests and jobs, limit retries, and give every loop a clear end condition.

Example: The ERP has 40,000 products, but normally only 300 change during the day. Instead of processing all 40,000 every 15 minutes, the integration requests only the products modified since the last successful sync.

3. Define and map the fields

List every field that must move between the systems. For each one, record its source field, destination field, data type, whether it is required, its default value, and any transformation and validation rules.

Here is an example mapping:

ERP fieldE-shop fieldRule
product_iderp_product_idStore as the external identifier
product_codeskuRequired and unique
product_namenameTrim surrounding whitespace
product_descriptiondescriptionPreserve supported formatting
retail_pricepriceConvert to the required decimal format
available_quantitystock_quantityDo not allow negative stock unless supported
vat_categorytax_classConvert using a predefined VAT mapping
is_activestatusConvert active or inactive to published or hidden
updated_aterp_updated_atUse for incremental sync

Do not assume that fields with similar names mean the same thing. The ERP field quantity may be physical warehouse stock, while the e-shop field stock_quantity may be the amount left for online sales after reservations.

Example: If the ERP reports 15 physical units but reserves 3 for existing orders, the integration may need to send 12 available units to the e-shop.

4. Define the source of truth

For every synced field, decide which system owns the value. Without a clear source of truth, the two systems may keep overwriting each other.

  • The ERP owns: Product codes, prices, stock, and VAT categories.
  • The e-shop owns: SEO titles, product slugs, and marketing content.

The integration should update only the fields owned by the source system.

Example: The ERP sends the product name, price, and stock. The e-shop administrator writes an SEO title and a longer marketing description. The sync must not overwrite those e-shop fields.

In a two-way integration, define ownership for each field separately. Do not treat both complete records as editable from either side.

5. Define how records are identified

Every source record must match the correct destination record. Use a stable unique identifier, such as an ERP product ID, an external system ID, an immutable SKU, or a UUID.

Do not match products by name. Names can change and may not be unique.

Store the ERP identifier in the e-shop whenever possible.

Example: ERP product 1872 is created in the e-shop and stores 1872 in its erp_product_id field. Future sync runs search by this identifier before creating or updating the product.

If a product code can change, do not use it as the only permanent link between the systems. Use the internal ERP identifier instead.

6. Choose how sync starts

The integration can start in several ways:

  • Scheduled polling: The integration asks the ERP for changes at regular intervals.
  • Webhooks: The ERP notifies the integration when an event happens. When reliable, this reduces unnecessary polling and gives faster updates.
  • A message queue: Work waits in a queue until it can be processed.
  • A manual action: An administrator starts the sync.

Webhook delivery can fail or be repeated. The integration must validate, store, and process webhook events safely.

A practical design may combine these methods:

  • Use webhooks for quick updates.
  • Run a scheduled reconciliation to catch anything the webhooks missed.
  • Give administrators a protected manual sync option.

Example: When a product changes, the ERP sends a product.updated webhook containing the product ID. The integration puts the ID in a queue. A background worker fetches the complete product from the ERP and updates the e-shop. Every night, a scheduled job checks all products changed during the previous day.

7. Design the sync workflow

Write down the complete sequence before you build it. A basic product sync may look like this:

  1. Start a sync run.
  2. Read the checkpoint, which marks where the last successful sync ended.
  3. Request modified products from the ERP.
  4. Validate each product.
  5. Transform ERP fields into the e-shop format.
  6. Find the matching e-shop product.
  7. Create or update the product.
  8. Record the result.
  9. Continue to the next batch.
  10. Save the new checkpoint after successful processing.
  11. Complete the sync run.

Keep data retrieval, validation, transformation, and destination updates as separate steps. Failures are then easier to find, and you can test each part on its own.

Example: A run starts with checkpoint 2026-07-12 08:00:00. The ERP returns 230 products modified after that time. The integration processes them in three batches and updates the checkpoint only after the required records are handled successfully.

8. Validate and transform the data

Do not send source data to the destination without checking it first.

Validate:

  • Format: Required fields, data types, string length, numeric ranges, and date formats.
  • Business values: Allowed values, currency, tax categories, and product status.
  • Links: Relationships between records.

Transform values when the two systems use different formats:

  • Text: Convert the encoding, trim whitespace, and remove unsupported HTML.
  • Other formats: Convert decimal separators, boolean values, and date formats.
  • Business values: Map VAT identifiers and apply default values.

Invalid data should produce a clear error. It should not silently create an incomplete or incorrect product.

Example: The ERP sends 24_PERCENT as the VAT category, while the e-shop expects the tax class ID standard-24. The integration uses a predefined mapping to convert the value before sending the product.

9. Secure the integration

Run sync processes on trusted backend infrastructure.

Do not expose ERP credentials, API keys, client secrets, or sync logic in browser JavaScript.

Anyone holding a valid bearer token may be able to use its permissions. Protect tokens and other credentials both in storage and during transmission.

Store secrets in environment variables, a secret-management service, or protected server configuration.

Also use:

  • HTTPS.
  • Short-lived tokens when supported.
  • The minimum API permissions required.
  • Credential rotation.
  • Server-side authorization.
  • Protected logs.
  • Network restrictions where practical.

If administrators can start a sync from a user interface, the browser should call a protected backend endpoint. The backend must verify that:

  • The user is signed in.
  • The session or access token is valid.
  • The user has permission to start a sync.
  • The request has not been forged.
  • No incompatible sync is already running.

Enforce access control in trusted server-side code. Do not rely on hiding parts of the interface or on client-side checks.

Example: The administration page shows a “Synchronize products” button only to authorized users. Pressing it sends an authenticated request to the backend. The backend checks the administrator permission and puts a sync job in the queue. The browser never receives the ERP API key.

For incoming webhooks, validate the provider's signature before processing the payload. A shared webhook secret can confirm that a request came from the expected provider and was not modified.

10. Control resource usage

Do not load or process the whole dataset in one operation when you can split it into batches.

Use API and database pagination, limited batch sizes, limited parallel processing, memory limits, request and job timeouts, and rate-limit handling.

Batching can reduce the number of operations and improve efficiency when the API supports batch actions.

Every loop needs a clear end condition. For paginated APIs, verify that:

  • The next-page cursor changes.
  • Empty pages stop processing.
  • The maximum page count is reasonable.
  • Repeated cursors are detected.
  • Failed pages can be retried safely.

Set timeouts so a failed network request or a faulty loop cannot run forever. Job-level timeouts can stop work that gets stuck or runs much longer than expected.

Example: The integration requests 100 ERP products per page and processes two batches at once. Each API request has a 30-second timeout, and the complete sync job has a 30-minute limit.

11. Prevent duplicates with idempotency

The same product update may be processed more than once. Manual retries, network timeouts, repeated webhook deliveries, queue redelivery, worker restarts, and uncertain API responses can all cause this.

Idempotency means a failed request can be retried without repeating its intended effect. Processing the same event twice must not create two products or produce an incorrect result.

To achieve this, use:

  • Stable external identifiers.
  • Upsert operations, which update a record if it exists and create it if it does not.
  • Unique database constraints.
  • Records of processed events.
  • Idempotency keys.
  • Version or timestamp checks.

Message systems may deliver the same message more than once. Background handlers should recognize repeated messages and process them safely.

Example: The ERP sends the same product.updated event twice for product 1872. The integration finds the e-shop product through erp_product_id = 1872 and updates it instead of creating another product.

12. Handle failures and retries

Not every failure should be retried. Sort failures into categories.

Validation errors

Examples include a missing SKU, an invalid price, or an unknown VAT category.

These usually need a data correction. Repeating the same request with the same data will not solve the problem.

Authentication and authorization errors

Examples include expired credentials, an invalid API key, or a missing permission.

Stop or pause the process and notify an administrator.

Temporary failures

Examples include a network timeout, a temporarily unavailable service, HTTP 500, 502, 503, or 504, and temporary rate limiting.

Retry these with a maximum number of attempts, increasing delays, random jitter, and a final failed state.

Exponential backoff increases the wait between retries. Jitter stops many workers from retrying at exactly the same moment. Retries should always have a maximum limit.

If an API returns Retry-After or rate-limit reset information, respect those values before sending another request.

Example: Updating a product returns HTTP 503. The integration retries after 2 seconds, then 5 seconds, then 12 seconds. After the last failed attempt, it records the product as failed and continues with the remaining products.

13. Handle partial failures

A sync run may update some products and fail on others. Do not treat the whole run as if nothing happened.

Record the outcome of each item: created, updated, skipped, unchanged, invalid, failed, or waiting for retry.

Decide whether the run should:

  • Continue after one product fails.
  • Stop immediately.
  • Retry only the failed products.
  • Roll back completed changes when possible.

For product sync, continuing is usually more practical than stopping the whole run because one product has invalid data.

Example: A batch contains 100 products, and 98 sync successfully. One has an unknown VAT code, and one API request times out. The integration records the first as invalid, schedules the second for retry, and keeps the 98 successful updates.

14. Define how deletions and inactive products work

Decide what happens when a product is deleted or disabled in the ERP. You could permanently delete it from the e-shop, hide it from customers, mark it inactive, set its stock to zero, archive it, or take no automatic action.

Permanent deletion may also remove product URLs, images, reviews, order references, SEO information, and historical data.

In many cases, disabling or archiving a product is safer than deleting it automatically.

Example: When an ERP product becomes inactive, the integration sets the e-shop product status to hidden and its stock to zero. It does not permanently delete the product.

15. Add logging and monitoring

Every sync run should leave enough information to show what happened.

For each run, record:

  • Details: Run identifier, trigger type, start time, end time, duration, and checkpoint.
  • Results: Number of records retrieved, created, updated, skipped, and failed.
  • Problems: Retry attempts and general error messages.

For each failed item, record the external product identifier, processing step, error category, error message, and retry status.

Do not log passwords, API keys, access tokens, complete authentication headers, or unnecessary personal information.

Add alerts for important conditions:

  • Sync has not run recently.
  • The failure rate is above an acceptable level.
  • Credentials have expired.
  • API rate limits are repeatedly reached.
  • A job has run longer than expected.
  • The queue holds too many unprocessed items.

Example: A sync dashboard shows that run SYNC-20260712-1045 retrieved 300 products, created 12, updated 270, skipped 15, and failed 3. An administrator can inspect the three failed product identifiers and their errors.

16. Test the integration

Test each part on its own, then test the complete workflow.

  • Data: Field mapping, validation, and transformation.
  • Access: Authentication and authorization.
  • Normal behavior: API contracts, creating and updating products, and manual sync.
  • Failure cases: Duplicate delivery, retries, timeouts, rate limits, and partial failures.
  • Scale: Large datasets.

Use test or sandbox environments when the connected systems provide them. Do not test destructive operations on production data without safeguards.

Example: Send the same product event twice and verify that only one e-shop product exists. Then simulate an e-shop timeout and confirm that the retry updates the same product instead of creating a duplicate.

17. Deploy and operate the integration

Decide where the integration will run. Options include the ERP backend, the e-shop backend, a separate integration service, a serverless function, a scheduled worker, or a queue worker.

Prevent incompatible sync runs from overlapping. Distributed locks, unique active-job constraints, queue-level deduplication, and run-status checks can all help.

Keep configuration separate from code. This may include API endpoints, credentials, batch sizes, timeouts, retry limits, sync frequency, field mappings, and feature switches.

Example: A scheduler starts product sync every 15 minutes. Before starting, the worker checks whether another product sync is active. If one is still running, the new run is skipped or delayed.

18. Document and maintain the integration

Document:

  • Purpose: The goal, source and destination systems, source-of-truth decisions, and field mappings.
  • Connection: Authentication method, API endpoints, trigger methods, and scheduling.
  • Behavior: Batch size, timeout values, retry rules, and error handling.
  • Operations: Deployment process, monitoring, recovery procedures, manual sync procedure, and known limitations.

APIs and business requirements change over time. Review the integration when:

  • An API version, field name, or authentication method changes.
  • Product volume increases or new product types appear.
  • Business ownership of a field changes.
  • Sync becomes slower or failure rates increase.

Example: The ERP introduces multiple warehouses. The documentation must then explain whether the e-shop receives total stock, stock from one warehouse, or stock calculated with a new availability rule.

A reliable API integration is more than a script that moves data. It is a controlled process with clear ownership, secure execution, predictable failure handling, measurable results, and documentation that lets another developer understand and maintain it.

Sources and further reading

The scenarios and identifiers in this article are illustrative. Production behavior must be validated against the documentation, limits, security model, and failure semantics of the systems being connected.