[ixpmanager] RADB WHOIS Rate Limiting

[Richard Laager]

For now, I am using the time honored hack of a sleep(). I'm not sure if 
the runInBackground() is necessary, but since the sleep()s make this 
take substantially longer, that seemed like a reasonable thing to try.

Any thoughts on a better answer?

diff --git a/app/Console/Commands/Irrdb/UpdateAsnDb.php 
b/app/Console/Commands/Irrdb/UpdateAsnDb.php
index 5332a0ec4..a91e76ead 100644
--- a/app/Console/Commands/Irrdb/UpdateAsnDb.php
+++ b/app/Console/Commands/Irrdb/UpdateAsnDb.php
@@ -67,7 +67,13 @@ class UpdateAsnDb extends UpdateDb

          $customers = $this->resolveCustomers();

+        $first = true;
          foreach( $customers as $c ) {
+            if ($first) {
+                $first = false;
+            } else {
+                sleep(1);
+            }
              $task = new UpdateAsnDbTask( $c );
              $this->printResults( $c, $task->update(), 'asn' );
          }
diff --git a/app/Console/Commands/Irrdb/UpdatePrefixDb.php 
b/app/Console/Commands/Irrdb/UpdatePrefixDb.php
index d75a892c4..dc5fbd847 100644
--- a/app/Console/Commands/Irrdb/UpdatePrefixDb.php
+++ b/app/Console/Commands/Irrdb/UpdatePrefixDb.php
@@ -67,7 +67,13 @@ class UpdatePrefixDb extends UpdateDb

          $customers = $this->resolveCustomers();

+        $first = true;
          foreach( $customers as $c ) {
+            if ($first) {
+                $first = false;
+            } else {
+                sleep(1);
+            }
              $task = new UpdatePrefixDbTask( $c );
              $this->printResults( $c, $task->update(), 'prefix' );
          }
diff --git a/app/Console/Kernel.php b/app/Console/Kernel.php
index f00333073..b7ca9b90c 100644
--- a/app/Console/Kernel.php
+++ b/app/Console/Kernel.php
@@ -57,9 +57,11 @@ class Kernel extends ConsoleKernel
          // IRRDB - https://docs.ixpmanager.org/features/irrdb/
          if( config( 'ixp.irrdb.bgpq3.path' ) && is_executable( config( 
'ixp.irrdb.bgpq3.path' ) ) ) {
              $schedule->command( 'irrdb:update-prefix-db' )->cron( '7 
*/6 * * *' )
+                ->runInBackground()
                  ->skip( function() { return env( 
'TASK_SCHEDULER_SKIP_IRRDB_UPDATE_PREFIX_DB', false ); } );

              $schedule->command( 'irrdb:update-asn-db' )->cron( '37 */6 
* * *' )
+                ->runInBackground()
                  ->skip( function() { return env( 
'TASK_SCHEDULER_SKIP_IRRDB_UPDATE_ASN_DB', false ); } );
          }



On 2023-06-07 00:20, Richard Laager via ixpmanager wrote:
>
> I seem to be getting rate limited by RADB with update-asn-db / 
> update-prefix-db. It will get partway through and then die. Doubly 
> unfortunate is the fact that IXP Manager doesn't keep track of where 
> it was in the cache run, so the next time it starts over again at the 
> top. As a result, networks farther down probably never get updated.
>
> Are other people running into this?
>
>
> https://www.radb.net/support/informational/query.html says:
>
>> Please note: The RADb WHOIS service is rate-limited. For large 
>> queries, we recommend using the RADb API
>>
>> Programmatic access is considered abuse of the service.
>
> Isn't that exactly what IXP Manager / bgpq3 / bgpq4 are doing???
>
>
> Later, it says:
>
>> If you plan on making a large number of queries please invoke a 
>> persistent TCP/IP session. This is done by telnetting directly to 
>> whois.radb.net and issuing the !! command. This will spare our server 
>> having to establish and teardown connections for every query.
>
> bgpq4, at least, does that. But IXP Manager is still separately 
> invoking bgpq4 for each AS when running update-asn-db / update-prefix-db.

-- 
Richard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.inex.ie/pipermail/ixpmanager/attachments/20230611/ee4f24f8/attachment.htm>