<html theme="themeKey88rand710" iconset="monochrome"><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head><body text="#000000"><br>
Just so I'm clear:<br>
<br>
- works when run manually on the command line for all members?<br>
- does not work when executed via artisan scheduler?<br>
<br>
How do you know it's not working? You've said you have no logs, etc. so
just trying to understand what "not working" looks like.<br>
<br>
You can lose the '
>/dev/null 2>&1' from the the artisan cron and see what
messages crop up? <br>
<br>
You can change the schedule in app/Console/Kernel.php for testing purposes.<br>
<br>
How many members are you running this for? There's IXPs using bgpq3
against radb for hundreds of members, including some really big ones.
Are you running any other queries against radb separate to this that may
cause them to limit your IP?<br>
<br>
- Barry<br>
<br>
<span>
</span><br>
<blockquote type="cite"
cite="mid:60e5ba10-76ca-bf9d-05df-17f210b69e48@wiktel.com"
style="border: 0px none ! important;">
<div xmlns="http://www.w3.org/1999/xhtml" class="__pbConvHr"
style="margin:30px 25px 10px 25px;"><div
style="width:100%;border-top:2px solid
rgba(146,154,163,0.7);padding-top:10px;"> <div
style="display:inline-block;white-space:nowrap;vertical-align:middle;width:49%;">
<a style="color:#485664
!important;padding-right:6px;font-weight:500;text-decoration:none
!important;" href="mailto:ixpmanager@inex.ie" moz-do-not-send="true">Richard
Laager via ixpmanager</a></div> <div
style="display:inline-block;white-space:nowrap;vertical-align:middle;width:48%;text-align:
right;"> <font color="#909AA4"><span style="padding-left:6px">16
June 2023 at 05:33</span></font></div> </div></div>
<div xmlns="http://www.w3.org/1999/xhtml" class="__pbConvBody"
__pbrmquotes="true"
style="color:#909AA4;margin-left:24px;margin-right:24px;">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<div class="moz-cite-prefix">My IRRDB updates are definitely not
working. They work when run manually, though, whether through the
web interface or the CLI (for one AS or for everyone). When I
looked before, I saw that the Laravel scheduler runs them with
>/dev/null 2>&1, so I get nothing in the output there.</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Should there be logs somewhere? I have
nothing in storage/logs/laravel.log.</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Thoughts on next troubleshooting steps?</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">I backed out my runInBackground()
change, so we'll see if that makes a difference.<br>
</div>
<div>-- <br>Richard</div>
<br><fieldset class="mimeAttachmentHeader"></fieldset><br><div>_______________________________________________<br>INEX
IXP Manager mailing list<br><a class="moz-txt-link-abbreviated" href="mailto:ixpmanager@inex.ie">ixpmanager@inex.ie</a><br>Unsubscribe or change
options here: <a class="moz-txt-link-freetext" href="https://www.inex.ie/mailman/listinfo/ixpmanager">https://www.inex.ie/mailman/listinfo/ixpmanager</a><br></div>
</div>
<div xmlns="http://www.w3.org/1999/xhtml" class="__pbConvHr"
style="margin:30px 25px 10px 25px;"><div
style="width:100%;border-top:2px solid
rgba(146,154,163,0.7);padding-top:10px;"> <div
style="display:inline-block;white-space:nowrap;vertical-align:middle;width:49%;">
<a style="color:#485664
!important;padding-right:6px;font-weight:500;text-decoration:none
!important;" href="mailto:ixpmanager@inex.ie" moz-do-not-send="true">Richard
Laager via ixpmanager</a></div> <div
style="display:inline-block;white-space:nowrap;vertical-align:middle;width:48%;text-align:
right;"> <font color="#909AA4"><span style="padding-left:6px">12
June 2023 at 04:27</span></font></div> </div></div>
<div xmlns="http://www.w3.org/1999/xhtml" class="__pbConvBody"
__pbrmquotes="true"
style="color:#909AA4;margin-left:24px;margin-right:24px;">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<div class="moz-cite-prefix">For now, I am using the time honored
hack of a sleep(). I'm not sure if the runInBackground() is
necessary, but since the sleep()s make this take substantially
longer, that seemed like a reasonable thing to try.</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Any thoughts on a better answer?<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix"><font face="monospace">diff --git
a/app/Console/Commands/Irrdb/UpdateAsnDb.php
b/app/Console/Commands/Irrdb/UpdateAsnDb.php<br>
index 5332a0ec4..a91e76ead 100644<br>
--- a/app/Console/Commands/Irrdb/UpdateAsnDb.php<br>
+++ b/app/Console/Commands/Irrdb/UpdateAsnDb.php<br>
@@ -67,7 +67,13 @@ class UpdateAsnDb extends UpdateDb<br>
<br>
$customers = $this->resolveCustomers();<br>
<br>
+ $first = true;<br>
foreach( $customers as $c ) {<br>
+ if ($first) {<br>
+ $first = false;<br>
+ } else {<br>
+ sleep(1);<br>
+ }<br>
$task = new UpdateAsnDbTask( $c );<br>
$this->printResults( $c, $task->update(),
'asn' );<br>
}<br>
diff --git a/app/Console/Commands/Irrdb/UpdatePrefixDb.php
b/app/Console/Commands/Irrdb/UpdatePrefixDb.php<br>
index d75a892c4..dc5fbd847 100644<br>
--- a/app/Console/Commands/Irrdb/UpdatePrefixDb.php<br>
+++ b/app/Console/Commands/Irrdb/UpdatePrefixDb.php<br>
@@ -67,7 +67,13 @@ class UpdatePrefixDb extends UpdateDb<br>
<br>
$customers = $this->resolveCustomers();<br>
<br>
+ $first = true;<br>
foreach( $customers as $c ) {<br>
+ if ($first) {<br>
+ $first = false;<br>
+ } else {<br>
+ sleep(1);<br>
+ }<br>
$task = new UpdatePrefixDbTask( $c );<br>
$this->printResults( $c, $task->update(),
'prefix' );<br>
}<br>
diff --git a/app/Console/Kernel.php b/app/Console/Kernel.php<br>
index f00333073..b7ca9b90c 100644<br>
--- a/app/Console/Kernel.php<br>
+++ b/app/Console/Kernel.php<br>
@@ -57,9 +57,11 @@ class Kernel extends ConsoleKernel<br>
// IRRDB - <a class="moz-txt-link-freetext"
href="https://docs.ixpmanager.org/features/irrdb/"
moz-do-not-send="true">https://docs.ixpmanager.org/features/irrdb/</a><br>
if( config( 'ixp.irrdb.bgpq3.path' ) &&
is_executable( config( 'ixp.irrdb.bgpq3.path' ) ) ) {<br>
$schedule->command( 'irrdb:update-prefix-db'
)->cron( '7 */6 * * *' )<br>
+ ->runInBackground()<br>
->skip( function() { return env(
'TASK_SCHEDULER_SKIP_IRRDB_UPDATE_PREFIX_DB', false ); } );<br>
<br>
$schedule->command( 'irrdb:update-asn-db'
)->cron( '37 */6 * * *' )<br>
+ ->runInBackground()<br>
->skip( function() { return env(
'TASK_SCHEDULER_SKIP_IRRDB_UPDATE_ASN_DB', false ); } );<br>
}<br>
<br>
</font><br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">On 2023-06-07 00:20, Richard Laager via
ixpmanager wrote:<br>
</div>
<div>-- <br>Richard</div>
<br><fieldset class="mimeAttachmentHeader"></fieldset><br><div>_______________________________________________<br>INEX
IXP Manager mailing list<br><a class="moz-txt-link-abbreviated" href="mailto:ixpmanager@inex.ie">ixpmanager@inex.ie</a><br>Unsubscribe or change
options here: <a class="moz-txt-link-freetext" href="https://www.inex.ie/mailman/listinfo/ixpmanager">https://www.inex.ie/mailman/listinfo/ixpmanager</a><br></div>
</div>
<div xmlns="http://www.w3.org/1999/xhtml" class="__pbConvHr"
style="margin:30px 25px 10px 25px;"><div
style="width:100%;border-top:2px solid
rgba(146,154,163,0.7);padding-top:10px;"> <div
style="display:inline-block;white-space:nowrap;vertical-align:middle;width:49%;">
<a style="color:#485664
!important;padding-right:6px;font-weight:500;text-decoration:none
!important;" href="mailto:ixpmanager@inex.ie" moz-do-not-send="true">Richard
Laager via ixpmanager</a></div> <div
style="display:inline-block;white-space:nowrap;vertical-align:middle;width:48%;text-align:
right;"> <font color="#909AA4"><span style="padding-left:6px">7
June 2023 at 06:20</span></font></div> </div></div>
<div xmlns="http://www.w3.org/1999/xhtml" class="__pbConvBody"
__pbrmquotes="true"
style="color:#909AA4;margin-left:24px;margin-right:24px;">
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<p>I seem to be getting rate limited by RADB with update-asn-db /
update-prefix-db. It will get partway through and then die. Doubly
unfortunate is the fact that IXP Manager doesn't keep track of
where it was in the cache run, so the next time it starts over
again at the top. As a result, networks farther down probably
never get updated.</p>
<p>Are other people running into this?<br>
</p>
<p><br>
</p>
<p><a moz-do-not-send="true"
href="https://www.radb.net/support/informational/query.html"
class="moz-txt-link-freetext">https://www.radb.net/support/informational/query.html</a>
says:<br>
<br>
</p><blockquote type="cite">Please note: The RADb WHOIS service is
rate-limited. For large queries, we recommend using the RADb API<br>
<br>
Programmatic access is considered abuse of the service.</blockquote>
<br>
Isn't that exactly what IXP Manager / bgpq3 / bgpq4 are doing???
<p><br>
Later, it says:<br>
<br>
</p><blockquote type="cite"><span style="white-space: pre-wrap; display: block; width: 98vw;">If you plan on making a large number of queries please invoke a persistent TCP/IP session. This is done by telnetting directly to whois.radb.net and issuing the !! command. This will spare our server having to establish and teardown connections for every query.</span></blockquote>
<br>
bgpq4, at least, does that. But IXP Manager is still separately
invoking bgpq4 for each AS when running update-asn-db /
update-prefix-db.
<pre class="moz-signature" cols="72">--
Richard</pre>
<br><fieldset class="mimeAttachmentHeader"></fieldset><br><pre wrap="">_______________________________________________
INEX IXP Manager mailing list
<a class="moz-txt-link-abbreviated" href="mailto:ixpmanager@inex.ie">ixpmanager@inex.ie</a>
Unsubscribe or change options here: <a class="moz-txt-link-freetext" href="https://www.inex.ie/mailman/listinfo/ixpmanager">https://www.inex.ie/mailman/listinfo/ixpmanager</a>
</pre>
</div>
</blockquote>
<br>
<div class="moz-signature">-- <br><br>
Kind regards,
<br>
Barry O'Donovan
<br>
INEX Operations<br>
<br>
<a class="moz-txt-link-freetext" href="https://www.inex.ie/support/">https://www.inex.ie/support/</a>
<br>
+353 1 531 3339<br>
<br>
<br>
</div>
</body></html>