Kernel Panic

srautanesrautane Registered User, Hog Beta
Today I was copying a cue or just going to cue I copied...not quite sure.

Playback was frozen suddenly and I noticed the yellow mark in right corner. I went to DP and it had kernel panic error...I was going to take a pic, but it started booting and I think, it loaded and installed firmware again. The whole thing took about 2-3min and after that the DP was up and running again.

I'm just wondering what can cause this...will it happen again?

Comments

  • jxgriffijxgriffi Registered User, DL Beta, Hog Beta
    edited September 2010
    Sami,

    Interesting you posted this. I had something similar happen on Sunday night to me. Running cues live (no programming at all) and I lost all my backlights (Studio Spots/Colors), but the front light kept going (they are stationary). I looked down and saw the yellow "!" on the console. Bent down to look at the DP8K and I quickly saw "....attack...." and couldn't catch everything else. I did a quick hard reset and the DP came back just fine and continued on the rest of the night with no issues.

    I chalked it up to an ethernet glitch of some type (I'm in a permanent install)..but since it happened to you too, I had to drop my two cents in.
  • cmuenchowcmuenchow Registered User, Administrator, HES Staff
    edited September 2010
    Kernel panics, to my knowledge, have never been reported in the field up to this point so this is the first I have heard of this happening in the field during normal playback. I would like to get copies of both Sami and Jon's show that was running at the time. I might be able to induce the error through overnight testing or just by getting the timing right.
  • jxgriffijxgriffi Registered User, DL Beta, Hog Beta
    edited September 2010
    Chris...

    I'll send my show when I get in this morning.

    Details (typical for a Sunday run):
    Console started up at 7am
    Show run at 9am-1045am and 1115am-115pm
    Sits idle from 130p till 4pm
    Show run from 5p-7p
    Console shut down around 7:30p

    "attack" happened approximately 38 minutes into the 5p show run.
  • srautanesrautane Registered User, Hog Beta
    edited September 2010
    I will send my show tomorrow, when I'm back to work.

    Our DP is installed in a theater and has almost always power on (UPS). Sometimes we switch it off (during holidays longer than 2 days). So, it is usually on Mon-Sun 24h per day.

    The DP has worked almost perfectly, but three times I have catched it booting itself.
    The first time it was doing it before I started the console, second time right after show load (It might had an error, but I didn't catch that, just saw it booting)
    Also, once it has had assert failed error -> boot.
    The kernel panic which is the latest "glitch" is the only one which has happened during programming/playback.
    We are running and rehearsing shows from 10am-10pm Mon->Sat
    Two shows are running and we are rehearsing one. All of them are based on the same file (I think Chris has some versions of the running ones already and also the "original")
    I will send the one we are rehearsing/programming right now.
  • cmuenchowcmuenchow Registered User, Administrator, HES Staff
    edited October 2010
    I think the common thread here is lengthy up time. I have several tests setup now to try and catch this. I am sure it will be random at best. The problem with long up time testing is that it takes a long time to diagnose. Rest assured we have this under our testing cycle here.
  • jxgriffijxgriffi Registered User, DL Beta, Hog Beta
    edited October 2010
    Thanks Chris...I'm in the same boat as Sami.

    My DP's never shut down (unless I have to do a hard reset). Console is up and down every night, but the DP's run constantly (along with the entire lighting network).

    Should I set it up as standard to shut down everything? It's no big deal...I just have to get the procedures in place and get everyone trained.
  • cmuenchowcmuenchow Registered User, Administrator, HES Staff
    edited October 2010
    If it is becoming a common problem for you Jon then, yes, I would recommend power-cycling the DP8000s every morning before you get started. According to our developers the cause may well have nothing to do with the application and require extensive digging through third party code to resolve. Right now we just need to establish a reliable way to reproduce the issue.
  • jxgriffijxgriffi Registered User, DL Beta, Hog Beta
    edited October 2010
    Well, we've had the DP8K for 2 years now I think...and MAYBE it's happened 3 times in that two year period (unsure because I never caught it before this last one). I'm not sure if there IS a reliable way to reproduce it...

    To err on the side of safety, I will implement a reboot of the DP with the console into the startup. (As long as they can remember to do it).

    That being said, our Sunday schedule wouldn't change (up at 7am and finally down at 9pm). That shouldn't be that big of a deal since we would have rebooted at 7am, correct (in theory of course)?
  • bradpepebradpepe Registered User, HES Alumni
    edited October 2010
    Jon,

    You said:
    To err on the side of safety, I will implement a reboot of the DP with the console into the startup. (As long as they can remember to do it).

    You could create a startup macro that resets the node(s)... Just keep in mind that the macro will fire after the DP loads the show and is ready. So it might confuse your staff when it is on line for a moment then offline, then back on line.
  • jxgriffijxgriffi Registered User, DL Beta, Hog Beta
    edited October 2010
    See...sometimes the easiest implementation eludes me.

    I didn't even THINK about doing the macro. I already have a comment macro for startup and I'll just add it to the end!

    Thanks Brad!
  • srautanesrautane Registered User, Hog Beta
    edited October 2010
    I think I will do that too.
    We have had our DP running since the beginning of August and like I said it had booted/crashed four times. But, nothing serious so far...shows are running fine. I will power cycle it regularly now.
  • cmuenchowcmuenchow Registered User, Administrator, HES Staff
    edited October 2010
    Sami and Jon,

    Are you running any networking on the fixture link port of the DP8K?
  • srautanesrautane Registered User, Hog Beta
    edited October 2010
    In my case everything is run via fixture link. All data is sent out as Art-Net and converted to DMX in Art-Net nodes.
    In Art-Net network I have switches, Art-Net nodes, one laptop (for monitoring) and two video projectors.
  • jxgriffijxgriffi Registered User, DL Beta, Hog Beta
    edited October 2010
    Same here...everything is run on fixture link. Leaving Art-net and going to Pathport nodes running Artnet.
  • srautanesrautane Registered User, Hog Beta
    edited October 2010
    Today, the kernel panic happened again...
    I did not power cycle after the previous panic, but did a node reset.
    This panic happened when I was just playing back a cue...but I'm not sure was I just pressing play or using goto.

    3ab1a573.jpg
  • cmuenchowcmuenchow Registered User, Administrator, HES Staff
    edited October 2010
    Excellent Sami. I will pass on to development.
Sign In or Register to comment.