Streaming buffer underrun error

Skydel Discussion
1

Hello,

We have suddenly run into a problem where the Skydel simulator (version: 21.9.1) consistently throws the error below:

2024-02-07 22:40:40.728358: <info> Simulator State: Error
2024-02-07 22:40:40.764746: <error> Streaming buffer underrun
2024-02-07 22:40:40.764839: <info> Simulator State: Ready
2024-02-07 22:40:40.834710: <info> Simulator State: Error
2024-02-07 22:40:40.841671: <error> Too many packets (more than 10000) arrived late in the radio. Stopping Simulation.

We have had the same setup for more than 2 years and never had this issue.

Unable to attach performance log but here is a plot of the entire run (Streamer thread):

performance
performance1187×719 12.6 KB

Also, the GPU speed test says “This GPU can perform real-time simulation”.

Has anyone seen this issue before? or any tips to figure out what is the root of the problem?

Thanks!
Surabhi

2

Attaching the GPU speed test results:

GPU test
GPU test802×549 71.5 KB

3

Hi @sgp,

Could you provide a screenshot of the performance graph if you can’t provide the log?
The streamer performance trace is not enough on it’s own to help me pin-point your issue, I would need the constellation and modulator trace as well.

Regards,
Mathieu

4

Hi @mathieu.favreau,

I initially did want to attach the performance.csv file but it seems like as new member/profile in this community I’m not allowed to attach any file other than images.
Is there another way (email?) I can send you the CSV file? or if you could allow me to upload it here that would be great. Thanks!
Surabhi

5

@sgp , you should be good to upload a CSV now. :smiley:

6

Hi @A-D
Thanks for looking into this but I still can’t seem to upload any attachments. I see the message “Sorry, new users can’t upload documents”.

7

Hi @sgp,

From the CSV @A-D shared with me, the issue is not with the radio or GPU, but with the system’s CPU.
I was able to import your file and look at it.

image
image861×404 17.6 KB

Since the blue line is going down, it’s the portion of the engine which runs on the CPU which is problematic.
Usually this ether means:

  • Something else is running on the machine which is taking the CPU resources from Skydel
  • The CPU is configured to Power Save mode and is being throttled by the OS
  • The CPU is getting too hot and is being throttled by the OS to prevent over-heating
  • Your system’s memory (RAM) is completely used and the OS is writing Skydel’s memory to Disk

I would recommend using HTOP if you are using Ubuntu to diagnose this issue.
You can install it using the command sudo apt install htop.
It will allow you to see which process is using your CPU, and how much memory is available.

image
image1204×262 55.6 KB

As you can see in this screenshot of HTOP, this system has a lot of memory available and the CPU is mostly unused. You can play with HTOP settings to show more information.

Also I would make sure your setup is in performance mode:

  • Ubuntu 18: In a terminal run the following command
    echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
  • Ubuntu 22: Go to power settings
    image
    image737×301 17.9 KB

Hope that helped,
Mathieu