[ixpmanager] SFLOW Under Reporting?

[Ian Chilton]

Hi,

> On 2023-06-27 00:31, Nick Hilliard (INEX) wrote:
> My initial suspicion would be tail drops due to a buffer overflow on
> the pipe between the sflowtool process and sflow-to-rrd-handler.

You beat to me to posting, but also came to this conclusion and I 
believe i've proved this to be the case.

I wrote this to simply parse the sflow data and sums it. Every 
'interval', it takes the total and zeros it. In a thread (which is 
irrelevant here but I was testing for the bigger workload), it prints it 
out.

https://gist.github.com/ichilton/b53cde596bb02289fca88fb61480c58f

It was 1AM when I did this and now only 07:30 so i've not tested it at 
peak traffic, but at lower traffic at those times i'm seeing between 
5-8% of the total traffic reported by MRTG (obviously that is over a 5 
minute interval and this is at a 1 minute interval, but it's in the 
ballpark).

So I believe, assuming it can even keep up normally, what is happening 
is while it's busy executing the periodic flush / mac table code, the 
buffer is overflowing and it's missing samples.

That explains why the shape/trend of the graphs are correct - they are 
lower in numbers than they should be.

A quick workaround would be to hack the current script to do the 
periodic flush/reload in a thread so it happens concurrently to the flow 
parsing.

Ultimately, we'll keep hitting this in the future as interfaces and 
traffic increases.

I plan to do the following:

- Re-write this in Go for better performance.

- I have an idea that we could have a thread per switch - each switch 
sends sflow to a different port, the script manages an sflowtool per 
switch, so that thread is only parsing a subset of the overall data (we 
have ~20 switches), which would make it more scalable.

- Nicolaas posted to the list about goflow2, which is worth comparing to 
sflowtool.

I'm busy for the rest of this week with travel/DCs/event and prep for 
that, but I plan to work on this more in the coming weeks.

In the meantime, any thoughts/ideas/suggestions are welcome.

Ian