Full BackUp

DeRudDeRud Registered User
Gents

when we see a full tracked backup solution ala GrandMa with the Hog III ??
This Year ??


Denis

Comments

  • teericksonteerickson Registered User, HES Alumni
    edited March 2006
    Denis,

    We have discussed the need for implementing functionality that would allow a seamless switch to backup hardware with no interruption to a running show if a hardware problem or software crash were to occur.

    We are still discussing the details of this implementation, so I can't estimate a timeline for when it will happen.

    The two obvious considerations are:

    1) How do we handle a console failure?
    2) How do we handle a node failure (DP2000 or IOP)?

    My answers would be:

    1) Run a server on the backup console that only listens for changes and keeps a current copy of the show file. If the primary console fails, allow this server to begin communicating with nodes.

    2) Allow a node to run in "backup mode", whereby it has the same net number as an active node and begin processing DMX and providing feedback if the primary active node with that net number goes offline.

    These are both high-level answers. There are a lot of details that need to be taken into consideration for this to work properly.

    How would you like this functionality to work? I can make sure you're opinion gets logged so that we can implement something that functions as our users would want it to.

    Thanks.
  • DeRudDeRud Registered User
    edited March 2006
    Ok

    I think we do 1 and 2.l We have two hardwares the Node and the console.
    So no hardware alone can do anything in a case of fire to get the show work perfect.


    denis
  • rosswillrosswill Registered User, Hog Beta
    edited March 2006
    I agree. I think both suggestions should be implemented.

    Only question I have is what happens if the original console/DP comes back to life. Are we left with a scenario where we have 2 servers or nodes conflicting?
  • DeRudDeRud Registered User
    edited March 2006
    HAY;

    i think if one cames alive this unit should be a client.
    Thats live : the early bird catch the day...lol

    Denis
  • HillbillyHillbilly Registered User
    edited March 2006
    Tom, you said
    "1) Run a server on the backup console that only listens for changes and keeps a current copy of the show file. If the primary console fails, allow this server to begin communicating with nodes."

    You used to be able to do this. Several years ago this was possible. For a time, you could run two servers on separate consoles and they would cleanly track each other. I was able to take either console on and offline with no conflicts. The only difference from what you suggested, is that both consoles would talk to the DP's at all times. I'd love to see this ability come back.

    Currently when I do a backup/tracking situation I set up it as a server/client situation. My server desk just sits there holding the show. I do all programming and playback on the client desk. This keeps most of the load off the server and let's it concentrate on running the show. This setup is not the best because you are still reliant on the server desk, and are in a world of hurt if it goes sideways. Having said that, the two desks will track each other when set up this way...plus you can do independent playback on both desks if wanted.

    I'm all for the tracking backup. That's been on my top 5 list since the beginning.
  • DeRudDeRud Registered User
    edited March 2006
    Hay Jon,

    the problem on this is that this solution is not nearly a BACKUP in a worse case of fire.
    We need very very soon a full tracked backup.
    So my ongoing think is that I link two console together with a midi cable.
    So I`f programming ends a load the show into the second III and it will
    track my cuelist in playback.
    So with this setup you have a full second system for backup.
    As you do this I have not good minds about it ( for BACKUP ) for others it is very very fine.
    So often I had crashes with server / client and all came down.

    Denis
  • rosswillrosswill Registered User, Hog Beta
    edited March 2006
    Hi Denis,

    I also run things in this way, however the problem comes when desk A goes offline the DP's reload the entire patch and go through their startup routine on connection to Desk B. They also do not hold the last dmx state all the way through the changeover process. Furthermore, the system is not truly backed up until the DP's have a spare also.

    Tom's suggestion is what is needed.

    Ross
  • DeRudDeRud Registered User
    edited March 2006
    Hay

    Yes I agree total.
    What I do next for a small cost backup solution is:
    When Hog III PC and the console have the same software on it ( 1.3.8 <> 1.4 ) then I conect a superwidget to my pc version and drive them.
    Cheaper then 2 consoles ; also for my client.
    I know that ´has a limit for 7 universes.

    Best
    Denis
  • Marty PostmaMarty Postma Registered User
    edited March 2006
    In the meantime.....I own a DMX DataLYNX unit that lives in my HOG-3 rack.....very handy little unit!!

    http://www.dmxdatalynx.com/

    They are distributed by TMB.


    {EDIT - This has the added advantage to C.Y.A. on the very rare occasion your DP takes a dump!}
  • teericksonteerickson Registered User, HES Alumni
    edited April 2006
    I've opened feature request #9212 for the fully tracking backup functionality that has been discussed here. If you have any great ideas or think of anything that's been missed, continue to post those thoughts to this thread and I'll continue adding comments to this bug.

    Thanks.
  • HgSiegenthalerHgSiegenthaler Registered User, Hog Beta
    edited April 2006
    Hi
    Can i somewhere on the web take a look on the request list?

    I'would like to have a full backup over my laptop with the hog pc!

    Cheers
    Hg
  • teericksonteerickson Registered User, HES Alumni
    edited April 2006
    Hg,

    We don't have a publicly searchable database of our request or bug lists. You can take a look through the Requests & Enhancements forum to see what people have requested there.

    As I mentioned above, we have enhancement #9212 open for the fully tracking backup functionality. Please post any additional ideas you have for that feature here and we can get them added to the bug.

    Thanks.
  • Akito.HAkito.H Registered User, Hog Beta
    edited April 2006
    I want to use Wholehog3 for the full backup.

    And, I want to monitor the state of each console from Hog3PC.

    I want to decide which console to use from Hog3PC of the monitor as a backup.

    As a result, it is possible to solve it immediately after the console can be observed by two or more numbers of people, and the problem occurs.

    Thanks,
    Akito
  • teericksonteerickson Registered User, HES Alumni
    edited April 2006
    Akito,

    Are you saying that you want to be able to manually switch which console is actively running the server?

    I can see how this might be useful. If I had to relocate my consoles from my tech position to my show position I could move the backup, manually switch to make that console the server, then move the other desk.
  • Akito.HAkito.H Registered User, Hog Beta
    edited April 2006
    Tom,
    Yes,
    I want to be able to manually switch which console is actively running the server.

    I wish to express my gratitude for your understanding.

    Akito
  • teericksonteerickson Registered User, HES Alumni
    edited April 2006
    Good idea, Akito.

    I've added that request to enhancement #9212.

    Thanks.
  • jankirchhoffjankirchhoff Registered User
    edited April 2006
    If you take a look and various HA (high availibility) or cluster projects going on for Webservers and Databases, you can learn a lot about how such things work in other software projects.

    Since a full backup system (hot spare) should automatically take over within less than a second, there should be at least 2 show servers up and running at the same time and both send their output to all DPs (kind of a redunant showserver cluster). The DP should ignore all packets coming from showserver 2 unless it doesn't get any data from showserver 1 any more.
    All clients in the show (Hog3s, IPCs, Hog3PC) should connect to all Showservers and send the same data to both of them. The showservers sync each other by exchanging all "events" they got from the clients in case something got lost.

    In the case of a crash of showserver 1 (running on a hog3 for example), I could still control the show from the desk of the other processes are still running and I have had a second showserver active in the network.
    Or I could pull the network cable of the hog3 and go on with the show on another desk without any interruption in the show or dmx.

    The second problem is the backup DP: We'd need to DPs doing exactly the same. If one crashes, just plug the dmx cables into the backup DP. This should not be configured at the desk but at the DP itself (i.e. I do not want to duplicate my patch on another 4 outputs)

    For the clients: There should be a slave mode like a MSC slave, but without all the MIDI stuff. We have ethernet here and my backup console might not be standing right next to the main console for whatever reason and I do not see any reason why there should be an additional cable be needed. Maybe just add a few network MIDI-Ports? So a console can send on "Network MIDI-Port 1" and another one listens on "Network MIDI-Port 1" in MSC slave mode?

    And although it doesn't really fit in here: The virtual DPs in Hog3PC should be available in the network just like every other DP, especially for these full backup strategies. So it should be possible to have the backup showserver running on a laptop and have superwidgets (or any mixture of usb-dmx-widgets and DP2000s) as backup DPs...

    just doing some brainstorming ;) hope it helps!
    Jan
  • teericksonteerickson Registered User, HES Alumni
    edited April 2006
    Jan,

    I'm not the architect here, but it seems to me like having both servers always sending traffic to all DPs would be a lot of unnecessary network traffic, especially since the DPs would be ignoring half of the traffic until they decided that the primary server failed. I think it would make more sense to have the DPs listen to all traffic and have the backup server make the determination that the primary has failed and then begin sending packets to the DPs.
  • jankirchhoffjankirchhoff Registered User
    edited April 2006
    You are safer when you make this decision on the receiving side. It will end up in a mess if showserver 2 thinks that showserver 1 is offline and starts sending data when showserver 1 is still active but there is only some packte loss/ sync problem inbeetween server 1 and 2.

    I never measured the amount of data the hog3 sends to the DPs, but one dmx-line is 250kbit per second and the hog3 does not send dmx but just commands to the DP since the DPs calculate fades themselves, so the amount of data should be less (but there is of course some overhead, so maybe its 250 to 300kbit per second again.). This would mean we could theoretically fit 300 Universes on a 100Mbit network, if we take some good switches and think about the network layout, even more will fit! 300 Universes would be 150 redunant ones. I think that is no problem. And it would make the whole configuration very stable and easy to handle (in terms of software development).

    But thats something your developers have to think about, I'm just sharing my personal experiences from the software projects I have been programming and I have been using... There are many theories and approaches, but I am always a big fan of very robust solutions. Better send too much data than nothing. Always assume that part of the network will fail because of a broken switch or something like that, assume some nodes will see each other and some other will only see half of them. And keep in mind that this always happens in a show while the operator is pushing a dozen flash-keys for ACLs and audience blinders. We do not want any "1 second delay for network reconfiguration" or something like that.

    Jan
  • teericksonteerickson Registered User, HES Alumni
    edited April 2006
    Jan,

    Why do you think it would be safer having the DP make the decision? You may very well be right, I just imagine that the decision is being made based on the network traffic that a device sees and that the secondary server could make the determination just as well as the DP could.
  • kmontagnekmontagne Registered User
    edited April 2006
    Tom, I too come from a large scale highly-available software design background. I believe the problem Jan is addressing by having the DP make the decision instead of the server is to avoid what is commonly referred to as a "split brain" situation. The challenge with having the servers make the decision is how to avoid the situation where they both think they are the "master/primary" and send packets confusing the receivers. In many software architectures this problem is addressed by having the "master/primary" exclusively acquire some resource. (For example, in database systems where data corruption can be catastrophic this exclusive resource can be a SCSI device or some other I/O fencing technique.)

    I would not really be able to suggest best alternative for the WHIII without fully knowing the underlying architecture. Depending on some of the design already there, combining your suggestion of letting the server decide and Jan's let the client DPs decide if they receive packets from multiple servers. Sometimes in these cases we make an arbitrary decision on which to listen to -- it might be based on the server with the lowest address/serial #/boot time/etc.

    I have a great deal of large scale high available architecture design experience. (I almost always seek to have no single points of failure [SPOFs] and no manual intervention required to minimize any human-introduced delay.) If you would like to talk more about the subject of availability feel free to contact me offline.
    _________________
    Kevin Montagne
    Litkam, Inc
    713-397-1930
    kevin (at) litkam.com
  • teericksonteerickson Registered User, HES Alumni
    edited April 2006
    Kevin,

    Thanks for the offer. Rest assured that among the developers here are some of the best architects I have ever worked with. I have no doubt they will be able to implement the proper solution for our needs.
  • pbeasleypbeasley Registered User
    edited April 2006
    Adding my 2cents....

    There needs to be some way of indicating to the human element that a transfer to the back-up has happened. Maybe even something as simple a a macro to execute when the server takes control, similar to, but seperate from the startup macro.

    This becomes especially important if you try to add live back-up options to systems that don't have an operator.
  • teericksonteerickson Registered User, HES Alumni
    edited April 2006
    That's a great point, Paul.
    I've added your comments to bug #9212.

    Thanks.
  • Akito.HAkito.H Registered User, Hog Beta
    edited April 2006
    Tom,

    I would like you to do the color of the frame of the window of the client and the server to the different one.

    As a result, even when two or more systems exist, which a server is easily distinguished.

    Thanks,
    Akito
  • Jan OpsethJan Opseth Registered User
    edited April 2006
    It should also be considerd that many clients are run "unmaned" aka Cruise ships etc. so there would be necisary to have an auto failover of the DMX also, I was thinking you made an "link" cable that connected both DP´s (prim/sec) and the secondary DP DONT output any power before any failover aqures! and then if so happends they auto switch? I dont now if this is physical possible with the current layout of the DP´s.

    The 2nd option, would be to make an "DP Merge" that is network connected and both DPs goes in to ( call it and advances merger) that auto change the DPs if one goes down, it could be controlled from the server side!

    regarding the last option someone would say it would be costly, but I would say the clients that REALLY need this would pay for a real redudant network ( even I would)


    r:finger:

    Edit: Sorry for all the misspellings!
  • teericksonteerickson Registered User, HES Alumni
    edited May 2006
    Akito,

    That's a great idea. I've added your comments to bug #9212.

    Jan,

    We will need to implement functionality to automatically switch to a backup DP if the primary fails, but the exact details will probably depend on the functional specification that isn't complete yet. It will need to support our existing hardware, so a DP-to-DP link cable probably isn't feasible. The DP Merge could be a possibility, but in addition to the costs of developing a new piece of hardward and the fact that you would need to purchase this for auto-switching, this would also introduce another point of possible failure into the system.
  • lbickfordlbickford Registered User
    edited August 2006
    Software: Hog III PC 1.4.1
    Console: Dell Inspiron 6400
    Single DMX Widget
    Hello,
    My company has recently purchased a single Hog widget and a PC to program and run our show on tour. Coming from a theatrical background, I have to say that I love the ease of programming that the Hog provides. Touring with the Hog has allowed me to walk into any French or Italian festival and have our show up and running in an hour, rather than have to re-program it into whatever console they may provide.
    That being said, the lack of stability of the Hog PC continues to worry me.
    The console crashed twice in programming during our recent tour. Thankfully, in both cases the festival dimmers "held" the dmx output.
    Now that we've returned home, I am looking into backup solutions for Hog PC.

    Based on the suggestions of Cat West at High End, I purchased two Midi-Sports and have "linked" two PC's running the same version of software and the same show. This works great, however, because I have only one widget, I can't get both laptops to recognize the widget. Has anyone figured out how to do this? Is it possible to use a usb switch so you can "choose" which laptop has control over the widget?

    Alternately, I am looking into a small dmx device such as the MicroTech DMX or Doug Fleenor's Preset 10 that could be plugged in-line between the PC and the widget. I could then record several dmx snapshots that could be used as backup looks if the PC console fails. Any suggestions or recommendations would be greatly appreciated.

    Thanks all. :hogsign:
    Laura Bickford
    Lighting Supervisor
    Bill T. Jones/Arnie Zane Dance Company
    lbickford@billtjones.org
  • z6p6tist6z6p6tist6 Registered User
    edited August 2006
    While I don't have anything to add to the implementation of the fail-over/backup system, I think it would also be good to consider what the actual DMX output is during fail-overs, saves, backups, etc.

    I would really like to have the option of setting a 'last look' or 'panic look' for the DP to catch if it goes into a reset. If possible, it would be nice to see the DP output a static 'panic look' or the last values output prior to the system rebooting.

    Example:

    If I'm in a rehearsal and I have to back up the show or reboot the console, the biggest hiccup for me is the stage lighting having a 'hiccup' or having to go dark. One way I avoid this is to record a panic look into a local DataLynx and activate that during system resets and backups.

    As the nodes become more distributed in larger systems, this becomes very difficult.

    It would be nice if I could tell the console/DPs to freeze a look during reboot or back-up.


    Bonus Prize: Have the console alert you with a pop-up (upon reboot) that there is a look frozen, with the options to cross-fade into the console's current state (with a fade time) or remain in the frozen look until a later time (if you still think there may be a problem at the console).

    Any thoughts?

    Phil
  • Marty PostmaMarty Postma Registered User
    edited August 2006
    I like your "Bonus Prize" idea phil.

    FYI - You can set a DataLynx to "Auto Backup". (under "Recieve Mode" - "Auto-Backup" "ENTER") Which means it will automatically keep the last known good DMX as Panic Scene #1 and activate it if DMX is lost. This helps if your DP and DataLynx live in a rack in dimmer beach for example. You do need to hit the "Exit" key to get out of this mode though.

    Also my experience has been that if a DP loses communication with the console it will "freeze" it's last known good DMX. It will drop this though once it's loading processes have finished once communication is restored. If you are able to fire off cues before the loading processes finish, the DMX will "jump" to the new look. There is no good way to time this, but it beats going to black when you don't want to.:hogsign:
  • z6p6tist6z6p6tist6 Registered User
    edited August 2006
    Thanks for the tip on the DataLynx Marty. I'll definitely make use of that.

    PG
  • Mike RedmerMike Redmer Registered User
    edited August 2006
    what is with crahses, which be a summary of work over a long time, a summary of commands, touches,...whatever? how will you handle this? you have the same hardware, same software, same showfile and at work all commands and operations you did behind the desk will send in realtime to a backupsystem and now? at this time, the backup system will help only if you have an electronical or mechanical problem. the most backup systems what we had actual in our business are be a real good fake and not 100% showsafe and will crash at the same time. So what is, when you have a delay to execute the same commands? your main system will crash, ok you go to the backup system and do the same, then? or you will not do your last actions behind your main system and do some other, but you will not know, from where you have this trouble, maybe 5 minutes erliear as you did some, what will in combination with some newer actions crash the console?

    i be happy that i must not write source code and think about this things, but in my opinion it is much safer two work with 2 identical but not connected main systems and use some dmx switches to change between the system. in this case at show time, you have an identical show as you started with the system, but the rest all commands, all operations, all... you did this on one of your systems, but on second you change only the pages.

    the most trouble with electronic equipment you will not have, when you play only or with your hardware equipment. the most trouble you will have with software, operators and in our business with patching, editing, programming,...(as a remember at introducing of windows and blue screens)

    so at first i think we need dmx switches, who can controlled over a network and you can cascade them, that you most not switch on/off 15 devices. in an advanced version with fade if the 2 inputs have different datas, or if you haven't a backup solution, then a frozen output and fade back if you have .... else switch;-)
  • DeRudDeRud Registered User
    edited August 2006
    IF HES are open for Artnet in future all can be done very easy !
    LanNode 4 did exact what a Hog III user need when ArtNet are in mind.
    DMX / artnet in > and then switch.

    But.............

    Denis
  • DeRudDeRud Registered User
    edited August 2006
    By all the way backup can be done:
    Have you all think in witch way you will backUp youre switches....lol



    Denis
  • GOGOGOGO Registered User
    edited August 2006
    Hi Denis,

    are you talking about the elc dmxlan node4?

    have you allready used it with WHIII and catalyst?

    also with Wyg?

    is it working?



    robert gurk
  • DeRudDeRud Registered User
    edited August 2006
    Hay Robert,

    first: NO.... I have never try this wile I playing with other stuff like
    pandoras and

    bringing the elc lan node 4 in the game was in mind for backup solution
    when HES / Hog III is open to Art-Net.
    As you know I own antoher console since....
    So use a console that output dmx and another one outputing at artnet and bring them into the elc...know you have a easy backup without cable; by the way like mike say: Use two ident. System and a DMX switch is the best.
    There is no way to give you 100% safty without cabling new.
    If the DMX switch fail....you have it again.
    What you do when you have more then 4 universe ?? you need more switches more more more more...add more things to a system rise also the sources of equipment that can fail......
    Normal 99% safty of a backup system ist fine enough.


    ....ArtNet is a nice way to manage backup systems.
    Yes yes you can make DMX ( DP ) to ArtNet...and then again you need more machines

    Denis
  • teericksonteerickson Registered User, HES Alumni
    edited August 2006
    You have all made some very good points, but there are two that I'd like to specifically highlight:

    1) As Mike mentioned, if you have a *full* tracking backup solution, your backup desk will be doing the exact same thing as the primary and can be affected by the same software instabilities. If you have an application problem that causes a crash on the primary desk it may have the exact same problem. Improving stability will not only help the console, but it will help the backup solution.

    2) Keep in mind that every piece of hardware that you add is an additional potential failure point. If you are converting from DMX to ArtNet and merging DMX universes, every converter, merger, DMX cable, and Ethernet cable is a link that can fail and cause a problem with your system. The more complicated your system is (even if it's made complicated to try to offer redundancy), the more places you can have problems.
  • DeRudDeRud Registered User
    edited August 2006
    Hay

    ..thats what I mean too.
    But:
    Without the Art-Net thing you have also rise the equipment.
    When you add a DMX switcher to the system you have also add a piece
    of equipment more...so and when you do this and rise the risk of fail
    you can also add a Art-Net node...smile

    I think that the backup solution only solve a hardware fail when it comes to
    the point of fallout.
    Software bug-backup can not be done full.
    I have a talk with some Network freeks weeks ago and say said:
    The main problem of system chrash is when people are on machines and typing keystrokes that can not be in mind of an software programmer.
    So having a peace of equipment in the network without give us the GUI
    to get in it runs safe on a high level ..like a network printer.

    Denis
  • teericksonteerickson Registered User, HES Alumni
    edited August 2006
    Denis,

    You are certainly correct. A device that's not running a gui and has very little user interaction is much less likely to fail than something like a lighting console.

    I don't tend to agree with the statement that people should have to think like a programmer to avoid a crash. A robust software application should be designed to be intuitive to the user and should be able to either handle or ignore incorrect syntax and input without crashing.

    Stop by the High End Systems trade show booths this fall and I may be able to give you some information about Wholehog 3 ArtNet processing.
Sign In or Register to comment.