Rereading Our Old Rakuten API and Amazon MWS Posts — How Design Changed in the SP-API Era

Aug 21
5 min read

Our blog still has posts from around 2020–2023: "How to Use the Rakuten Item Search API" and "Getting Started with Amazon MWS." The Rakuten API is still active, but MWS no longer exists. This time, let's revisit those two old posts and think about how we'd build things differently today. Hopefully useful for anyone building similar automation now.

API設計の大進化を示す図解。左側にある単純接続のシステムから、右側のAmazon SQSやLambdaと連携するサーバレスアーキテクチャへと進化する過程を描写。

The two posts we're revisiting

・How to Use the Rakuten Item Search API (around 2020): a simple setup — get an app ID, build search parameters, assemble a URL and send the request

・Getting Started with Amazon MWS (around 2020): authentication using an AWS access key and secret key, tested via the Scratchpad

Both were written from the same motivation: automating a seller's product research. Back then, the design was simple — just hitting an API once, directly.

MWS no longer exists

First, the premise: Amazon MWS (Marketplace Web Service) was fully retired on March 31, 2024. Reading the old MWS post today, the endpoints it describes no longer work.

The move from MWS to SP-API (Selling Partner API) was a major turning point — from the era when AWS IAM-signed requests (Signature Version 4) were mandatory, to a much simpler OAuth-based authentication. Building on SP-API today, none of the AWS access key / secret key steps from that old post are needed at all.

A quietly important lesson for writing technical posts, too: anything covering "authentication" is the first thing to go stale when a service's spec changes. In our own older posts, we've made a habit of adding a caveat note specifically around authentication sections.

The old design: hit the API once, directly

Both the Rakuten post and the MWS post shared the same design thinking at the time.

・Go fetch the info you want in a single, direct API call

・Don't worry much about rate limits — just call as needed

・Check the result on the spot, or at most log it

That was plenty for the use case back then (checking a few dozen products a month, manually researched). It was less "a program" and more "replacing point-and-click browser work with code."

Today's design: async, rate control, and floor protection — a set of three

The Amazon FBM (merchant-fulfilled) research and auto-pricing system we run today serves the same purpose — automatically fetching Amazon product data — but the architecture is completely different.

1. Put an SQS queue in between for async processing

We moved from "call directly when needed" to "queue it up, and let Lambda process it in order." SP-API has strict rate limits, so a direct-call design quickly runs into 429 errors (rate exceeded). Putting an SQS FIFO queue in front lets us control processing speed while working through a large number of ASINs.

📚 Related books

▶ AWSではじめるインフラ構築入門第2版 (Getting Started with AWS Infrastructure, 2nd Ed.)

Shoeisha ／ For learning the basics of combining AWS services like Lambda and SQS. A good foundation for understanding async designs like this one.

2. Retry with exponential backoff

429s still happen. Back then, we'd have just given up when an error occurred; now the design retries automatically. Instead of a fixed wait, exponential backoff doubles the wait time on each failure, letting the system calmly retry even under rate limiting.

# Skeleton of exponential backoff (shared across our SP-API calls in practice)
MAX_RETRIES = 10
wait = 2
for attempt in range(MAX_RETRIES):
    try:
        response = call_sp_api()
        return response
    except Exception as e:
        if attempt == MAX_RETRIES - 1:
            raise
        time.sleep(wait)
        wait = min(wait * 2, 60)

📚 Related books

▶ AWS運用入門改訂第2版 (Intro to AWS Operations, Revised 2nd Ed.)

Shoeisha ／ A book about the know-how of running things, not just launching them. Its take on error handling and cost management maps directly onto real operations.

3. Design price floors so they can't be broken

When building auto-pricing, the scariest failure mode is "a bug that keeps dropping the price with no limit." With the old single-call mindset, this kind of safeguard tended to get postponed.

In the current design, we read the seller-central floor price (minimum_seller_allowed_price) managed via the Amazon Listings Items API directly, and always clip the new price to that floor before updating it.

# Always clip to the floor price before updating
new_price = calculate_new_price(competitor_price)
floor_price = get_listing_min_price(sku)  # from Listings Items API
final_price = max(new_price, floor_price)
update_price(sku, final_price)

We considered maintaining the floor price in our own DB as well, but to avoid it drifting out of sync with Seller Central's value, we settled on treating Amazon's own data as the single source of truth. For SKUs with no floor set, we bulk-set an initial floor of 10% below the current price, and the program itself never changes the floor after that.

Why the design changed: because we got burned

Honestly, none of these three changes were obvious from the start. The story of DynamoDB costs spiking, and the one about 336,000 messages piling up in SQS — we've written about both on this blog before. The design only changed after we actually got hurt.

Looking back, the old design was also "correct for its time." At a few dozen items a month, a single direct API call is completely fine. The biggest cause of design rot is failing to notice a change in scale and dragging the same architecture forward. Once volume grows, it's worth considering the three — async, rate control, floor protection — earlier rather than later.

Another shift: from scraping to APIs

Not mentioned in the old posts, but another big change over these years is the data-fetching method itself. Places where we once used BeautifulSoup-based scraping are increasingly being replaced with officially sanctioned means, like SP-API or the Keepa API.

Scraping is easy to set up, but breaks easily when a site's structure changes, and on a platform like Amazon it carries the risk of account suspension. When an official API exists, using it — even if it takes more effort — pays off in lower operating cost over the long run.

Summary

Lining up the 2020 post against today's implementation, three things changed:

・A single direct call → async processing through an SQS queue

・Giving up on error → automatic retry with exponential backoff

・No price protection → always clip to Amazon's own floor price

A technical post starts going stale the moment it's published. Parts tied to "things the service controls" — authentication schemes, pricing structures — need revisiting every few years in particular. That said, there's no need to be embarrassed by an old post: keeping it as "a record of the best design at the time" turns it into something that shows your own growth when you look back.

If you need help designing or building an automation system on Amazon's SP-API, feel free to contact Robin Planning LLC. We can help with everything from rate-limit handling to price-protection mechanisms, drawing on lessons we learned the hard way.

📚 Related books

▶ AWS認定資格試験テキスト AWS認定クラウドプラクティショナー改訂第3版 (AWS Certified Cloud Practitioner Study Text, Revised 3rd Ed.)

SB Creative ／ For anyone who wants to relearn AWS systematically. Organizing your knowledge toward a certification also makes design calls like these easier.

* The links above are Amazon Associate links. Revenue from this blog goes toward running costs.

#SPAPI #Amazon #APIDesign #Lambda #SQS #Automation #Retrospective