hamburgscleanest/guzzle-advanced-throttle

A Guzzle middleware that can throttle requests according to (multiple) defined rules.

It is also possible to define a caching strategy, e.g. get the response from cache when the rate limit is exceeded or always get a cached value to spare your rate limits.

Using wildcards in host names is also supported.

Notice

You have to use the v2 branch if you are using Laravel or any Illuminate component below version 5.7.

Install

Via Composer

$ composer require hamburgscleanest/guzzle-advanced-throttle

Usage

General use

Let's say you wanted to implement the following rules:

20 requests every 1 seconds

100 requests every 2 minutes

1. First you have to define the rules in a hamburgscleanest\GuzzleAdvancedThrottle\RequestLimitRuleset:

$rules = new RequestLimitRuleset([
        'https://www.google.com' => [
            [
                'max_requests'     => 20,
                'request_interval' => 1
            ],
            [
                'max_requests'     => 100,
                'request_interval' => 120
            ]
        ]
    ]);

Make sure the host name does not end with a trailing slash. It should be https://www.google.com not https://www.google.com/.

2. Your handler stack might look like this:

 $stack = new HandlerStack();
 $stack->setHandler(new CurlHandler());

3. Push hamburgscleanest\GuzzleAdvancedThrottle\Middleware\ThrottleMiddleware to the stack.

It should always be the first middleware on the stack.

 $throttle = new ThrottleMiddleware($rules);

 // Invoke the middleware
 $stack->push($throttle());
 
 // OR: alternatively call the handle method directly
 $stack->push($throttle->handle());

5. Pass the stack to the client

$client = new Client(['base_uri' => 'https://www.google.com', 'handler' => $stack]);

Either the base_uri has to be the same as the defined host in the rules array or you have to request absolute URLs for the middleware to have an effect.

// relative
$response = $client->get('test');

// absolute
$response = $client->get('https://www.google.com/test');

Caching

Beforehand

Responses with an error status code 4xx or 5xx will not be cached (even with force-cache enabled)!

Available storage adapters

array (default)

Works out of the box. However it does not persist anything. This one will only work within the same scope. It's set as a default because it doesn't need extra configuration.

The recommended adapter is the laravel one.

laravel (Illuminate/Cache) - recommended

You need to provide a config (Illuminate\Config\Repository) for this adapter.

custom (Implements hamburgscleanest\GuzzleAdvancedThrottle\Cache\Interfaces\StorageInterface)

When you create a new implementation, pass the class name to the RequestLimitRuleset::create method. You'll also need to implement any sort of configuration parsing your instance needs. Please see LaravelAdapter for an example.

Usage:

$rules = new RequestLimitRuleset(
    [ ... ], 
    'force-cache', // caching strategy
    MyCustomAdapter::class // storage adapter
    );
    
$throttle = new ThrottleMiddleware($rules);

// Invoke the middleware
$stack->push($throttle());  

Laravel Drivers

General settings

These values can be set for every adapter.

    'cache' => [
        'ttl' => 900, // How long should responses be cached for (in seconds)?
        'allow_empty' => true // When this is set to false, empty responses won't be cached.
    ]

File

    'cache' => [
        'driver'  => 'file',
        'options' => [
            'path' => './cache'
        ],
        ...
    ]

Redis

    'cache' => [
        'driver'  => 'redis',
        'options' => [
            'database' => [
                'cluster' => false,
                'default' => [
                    'host'     => '127.0.0.1',
                    'port'     => 6379,
                    'database' => 0,
                ],
            ]
        ],
        ...
    ]

Memcached

    'cache' => [
        'driver'  => 'memcached',
        'options' => [
            'servers' => [
                [
                    'host'   => '127.0.0.1',
                    'port'   => 11211,
                    'weight' => 100,
                ],
            ]
        ],
        ...
    ]

Pass the config repository in the constructor of RequestLimitRuleset

$rules = new RequestLimitRuleset(
    [ ... ], 
    'cache', // caching strategy
    'laravel', // storage adapter
    new Repository(require '../config/laravel-guzzle-limiter.php') // config repository
    );

The same adapter will be used to store the internal request timers.

The adapters can be defined in the rule set.

$rules = new RequestLimitRuleset(
    [ ... ], 
    'cache', // caching strategy
    'array' // storage adapter
    );

Without caching - no-cache

Just throttle the requests. No caching is done. When the limit is exceeded, a 429 - Too Many Requests exception will be thrown.

$rules = new RequestLimitRuleset(
    [ ... ], 
    'no-cache', // caching strategy
    'array' // storage adapter
    );

With caching (default) - cache

Use cached responses when your defined rate limit is exceeded. The middleware will try to fallback to a cached response before throwing 429 - Too Many Requests.

$rules = new RequestLimitRuleset(
    [ ... ], 
    'cache', // caching strategy
    'array' // storage adapter
    );

With forced caching - force-cache

Always use cached responses when available to spare your rate limits. As long as there is a response in cache for the current request it will return the cached response. It will only actually send the request when it is not cached. If there is no cached response and the request limits are exceeded, it will throw 429 - Too Many Requests.

You might want to disable the caching of empty responses with this option (see General Driver Settings).

$rules = new RequestLimitRuleset(
    [ ... ], 
    'force-cache', // caching strategy
    'array' // storage adapter
    );

Custom caching strategy

Your custom caching strategy must implement CacheStrategy. It is suggested you use Cacheable for a parent class. This will give a good head start, see ForceCache and Cache for ideas.

To use your custom caching strategy, you'll need to pass the fully qualified cache name to RequestLimitRuleset.

Usage

$rules = new RequestLimitRuleset([ ... ], 
                                MyCustomCacheStrategy::class, 
                                'array', 
                                new Repository(...));
                                
$throttle = new ThrottleMiddleware($rules);
...                                

Wildcards

If you want to define the same rules for multiple different hosts, you can use wildcards. A possible use case can be subdomains:

$rules = new RequestLimitRuleset([
        'https://www.{subdomain}.mysite.com' => [
            [
                'max_requests'     => 50,
                'request_interval' => 2
            ]
        ]
    ]);

This host will match https://www.en.mysite.com, https://www.de.mysite.com, https://www.fr.mysite.com, etc.

Changes

Please see CHANGELOG for more information on what has changed recently.

Testing

$ composer test

Contributing

Please see CONTRIBUTING and CODE_OF_CONDUCT for details.

Security

If you discover any security related issues, please email chroma91@gmail.com instead of using the issue tracker.

Credits

• Timo Prüße
• Andre Biel
• All Contributors

License

The MIT License (MIT). Please see License File for more information.