Intermittent NodeBox Bug

I have come across what appears to be an intermittent NodeBox bug that becomes more likely the larger your network gets.

I managed to isolate the problem to a single innocent-looking subnetwork I made, Pick_Color, that simply chooses one of three colors based on a list of values. It does this by computing an index of 0 to 2 and rendering a switch node with the 3 color values as inputs.

The more this node appears in my network, the higher the probability of a crash.

By “crash” I mean a “Multiple entries with the same key” error which references one of my three color hex values. Although the error is not fatal, my NodeBox app stops working as soon as it appears and I have to quit and relaunch NodeBox. The error will appear the first time I run an animation or edit the network. But if it doesn’t appear the first time, it never will, and I can edit and run my network for the rest of my session without difficulty.

To demonstrate the remarkably steady probability with which this occurs, I made a very simple network (attached) which replicates the problem. It consists of a single Import CSV node which leads to 7 identical Pick_Color nodes. These in turn are joined to a single rendered Combine node.

The more Pick_Color nodes you connect to the Combine node, the higher the chance of an error the first time you attempt to “Edit the Children” of any one of the connected Pick_Color nodes. I verified this by launching this doc 100 times: 20 times with only 3 nodes connected, 20 times with 4 nodes connected, etc.

Here are my results:

3 Nodes: 15% Fail Rate
4 Nodes: 45% Fail Rate
5 Nodes: 60% Fail Rate
6 Nodes: 65% Fail Rate
7 Nodes: 70% Fail Rate

My current project needs to pick colors for a bunch of different value lists, so this bug is becoming an increasing problem for me.

Do you guys have any idea what’s going on here? Any suggested workarounds?

Thanks,

John

I am running NodeBox 3.0.43 with Java 1.8.0_25-b17 on a Yosemite MacBook Air. “Crash Test” network and CSV file attached.
Full typical error message copied below:

java.lang.IllegalArgumentException: Multiple entries with same key: <Node Red:color/color>=[#ff9966ff] and <Node Red:color/color>=[#ff9966ff]
at com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:150)
at com.google.common.collect.RegularImmutableMap.checkNoConflictInBucket(RegularImmutableMap.java:104)
at com.google.common.collect.RegularImmutableMap.<init>(RegularImmutableMap.java:97)
at com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:294)
at nodebox.node.NodeContext.<init>(Unknown Source)
at nodebox.client.NodeBoxDocument.render(Unknown Source)
at nodebox.client.NodeBoxDocument.requestRender(Unknown Source)
at nodebox.client.NodeBoxDocument.setActiveNetwork(Unknown Source)
at nodebox.client.NetworkView$GoInAction.actionPerformed(Unknown Source)
at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2022)
at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2346)
at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:402)
at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:259)
at javax.swing.AbstractButton.doClick(AbstractButton.java:376)
at javax.swing.plaf.basic.BasicMenuItemUI.doClick(BasicMenuItemUI.java:833)
at com.apple.laf.AquaMenuItemUI.doClick(AquaMenuItemUI.java:157)
at javax.swing.plaf.basic.BasicMenuItemUI$Handler.menuDragMouseReleased(BasicMenuItemUI.java:943)
at javax.swing.JMenuItem.fireMenuDragMouseReleased(JMenuItem.java:586)
at javax.swing.JMenuItem.processMenuDragMouseEvent(JMenuItem.java:483)
at javax.swing.JMenuItem.processMouseEvent(JMenuItem.java:429)
at javax.swing.MenuSelectionManager.processMouseEvent(MenuSelectionManager.java:329)
at javax.swing.plaf.basic.BasicPopupMenuUI$MouseGrabber.eventDispatched(BasicPopupMenuUI.java:870)
at java.awt.Toolkit$SelectiveAWTEventListener.eventDispatched(Toolkit.java:2425)
at java.awt.Toolkit$ToolkitEventMulticaster.eventDispatched(Toolkit.java:2317)
at java.awt.Toolkit$ToolkitEventMulticaster.eventDispatched(Toolkit.java:2316)
at java.awt.Toolkit$ToolkitEventMulticaster.eventDispatched(Toolkit.java:2316)
at java.awt.Toolkit.notifyAWTEventListeners(Toolkit.java:2275)
at java.awt.Component.dispatchEventImpl(Component.java:4777)
at java.awt.Container.dispatchEventImpl(Container.java:2292)
at java.awt.Component.dispatchEvent(Component.java:4703)
at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4898)
at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4533)
at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4462)
at java.awt.Container.dispatchEventImpl(Container.java:2278)
at java.awt.Window.dispatchEventImpl(Window.java:2739)
at java.awt.Component.dispatchEvent(Component.java:4703)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:746)
at java.awt.EventQueue.access$400(EventQueue.java:97)
at java.awt.EventQueue$3.run(EventQueue.java:697)
at java.awt.EventQueue$3.run(EventQueue.java:691)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:75)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:86)
at java.awt.EventQueue$4.run(EventQueue.java:719)
at java.awt.EventQueue$4.run(EventQueue.java:717)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:75)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:716)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:201)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)

Hello John,

thanks for the extensive bug report. We are looking into it.
Besides that we are wondering what you are trying to achieve. Whenever we see similar nodes we have a reflex: automation! I attach a NB3 file where I make the same thing you do to show the bug (I admit I'm guessing and it is unrelated to the bug) with only one subnetwork. The advantage of this is you can feed it with different lists of compare values and it outputs them as one list (and no crash). Note that the pick color node has in metadata the settings AND the list port set to LIST, not value.
regards,

Lucas

Hi Lucas,

Thanks for looking into this and thanks for thoughts on automation. I am still learning NodeBox and still learning how to make one node do the work of seven, so I really appreciate discussions like this.

The node you made may induce the same bug, but it does something different than I want to do. I don't want to vary the compare values. I have various lists of values coming in at various levels of my network, but I want them all color coded the same way: negative numbers red, values less than 7 yellow, and values greater than that green.

Why, then, do I need more than one color picker node? It's a valid question, and when I first tackled this project I tried to do just that. But this project is complex. It involves a series of different animations, each one involving an increasing number of dimensions, each with a similar but not identical left-right symmetry of comparison, each with an array of carefully sized, stacked, and colored boxes that fit together and deform in a precise way, each box filled with one or more labels that must adjust to the changing boxes in a responsive way, all of this driven by three separate CSV files, each filtered to multiple phases and requiring a half-dozen different lookups.

NodeBox does a brilliant job at handling all of this. But the chief problem I have with it is managing the complexity. This scaling problem is the central challenge not just for NodeBox but for visual languages in general. How to do you scale to hundreds of nodes and make connections across increasingly deep subnetwork hierarchies without producing an unfathomable pile of spaghetti?

After four or five failed attempts I finally hit on two basic techniques that helped me tame the beast. The first was farming out some of the messier calculations to Excel. By enriching the data in my CSV files just a bit I was able to greatly reduce the number crunching required by NodeBox. The trick here was learning what to farm out and what to leave in NodeBox. (A similar balance can be struck when farming things out to custom Python code.)

The second technique was to purposely duplicate certain subnetworks (like Pick_Color) and tuck them deeply away in various widely-separated corners of the network. This slightly increased the overall node count, and resulted in a somewhat less elegant, less automated network, but it made the overall structure much simpler and easier to modify. My top-level network is organized into cleanly separated clumps each with its own visible four-way symmetry, so whenever I need to make any change I know exactly where to go and which subnetworks to open.

These techniques are not needed when you only have a few dozen nodes. But when you have hundreds of nodes, some duplication makes it easier to encapsulate things.

My design is still in progress and evolving at a rapid pace (which is why I love using NodeBox: it lets me play and gives me immediate feedback at every step). Once it's done I may be able to go back and simplify some things, and arrive at a more concise and more "automated" overall structure with even less dependence on Excel.

But meanwhile this strange bug is an annoyance. Intermittent bugs are the worse. Everything was humming along fine until I added the third phase of my animation (initially by just copy-pasting the second phase). And even then nothing went wrong at first. It was quite puzzling the first time this error occurred, and even more puzzling when the exact same code sometimes produced the error and sometimes did not. My puzzlement turned to alarm when this initially rare problem began to occur more and more often.

I am less worried now that the problem seems to be confined to this one subnetwork. If need be I can try replacing it with a custom node or put these colors in the CSV files or start stringing extra wires across sections of my network to reduce the number of color_pickers needed. But it would be nice to know exactly what's going on and why this is happening. If it can happen with this node it may also occur in other situations.

Sorry for the long reply. Thanks for listening and for all the great work you guys are doing!

John

Any progress on this?

I just ran into it again in a different document, the generic morph node I just posted in the Show Your Work forum. I've launched that file dozens of time without a problem, but on two occasions this evening I got a similar error again complaining about duplicate color keys. When this happens I am unable to run any animations and have to quit and relaunch. Generic_Morph does not use my Pick_Color subnetwork, but does have numerous fill color lookups.

Error happened again on my morph document. This time I copied the error. Seems to have something to do with multiple colorize nodes but only happens every now and then...

java.lang.IllegalArgumentException: Multiple entries with same key: <Node colorize6:corevector/colorize>=[<Path>] and <Node colorize6:corevector/colorize>=[<Path>]
at com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:150)
at com.google.common.collect.RegularImmutableMap.checkNoConflictInBucket(RegularImmutableMap.java:104)
at com.google.common.collect.RegularImmutableMap.<init>(RegularImmutableMap.java:97)
at com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:294)
at nodebox.node.NodeContext.<init>(Unknown Source)
at nodebox.client.NodeBoxDocument.render(Unknown Source)
at nodebox.client.NodeBoxDocument.requestRender(Unknown Source)
at nodebox.client.NodeBoxDocument.setFrame(Unknown Source)
at nodebox.client.AnimationBar.stateChanged(Unknown Source)
at nodebox.ui.DraggableNumber.fireStateChanged(Unknown Source)
at nodebox.ui.DraggableNumber.mouseDragged(Unknown Source)
at java.awt.Component.processMouseMotionEvent(Component.java:6573)
at javax.swing.JComponent.processMouseMotionEvent(JComponent.java:3339)
at java.awt.Component.processEvent(Component.java:6294)
at java.awt.Container.processEvent(Container.java:2234)
at java.awt.Component.dispatchEventImpl(Component.java:4881)
at java.awt.Container.dispatchEventImpl(Container.java:2292)
at java.awt.Component.dispatchEvent(Component.java:4703)
at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4898)
at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4550)
at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4462)
at java.awt.Container.dispatchEventImpl(Container.java:2278)
at java.awt.Window.dispatchEventImpl(Window.java:2739)
at java.awt.Component.dispatchEvent(Component.java:4703)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:746)
at java.awt.EventQueue.access$400(EventQueue.java:97)
at java.awt.EventQueue$3.run(EventQueue.java:697)
at java.awt.EventQueue$3.run(EventQueue.java:691)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:75)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:86)
at java.awt.EventQueue$4.run(EventQueue.java:719)
at java.awt.EventQueue$4.run(EventQueue.java:717)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:75)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:716)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:201)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)

Any progress yet? Have you at least been able to replicate it?

Sorry to pester, but this thing is killing me. It just popped up again on a completely different project. As soon as my networks reach a certain size I start hitting this bug at random. I now have to test each file every time I open it by attempting to edit a subnetwork. If I get an error I have to quit and restart and repeat this until I can edit that first subnetwork. If I can, then I am fine from that point on.

In addition to slowing me down, this makes it embarrassing and impractical to demo my work. And above a certain size my projects stop working altogether.

The one common factor is that the error always mentions color in one way or another. So the bug must be hiding somewhere in or near some of your color-related code.

Thanks. Hope you can swat this soon!

John

Hello John,

you do not pester…
I’m pestering the people here and we do know what is going on, it is a little complicated.
However: the problem occurs when you duplicate nodes within different subnetworks that still carry the same name. If you have a colorize1 node in one subnetwork this same name should not be occurring in another subnetwork. Stefan claims that if they all have different names it will not occur.
Hope this helps, we are looking how we can avoid this in the future,

best,

Lucas

Lucas,

Thank you. This is a HUGE help.

For my larger projects I have been building modules that go four or five levels deep for each phase of animation and then copy-pasting as a starting point for each new phase. I am also building a library of standard subnetworks that I copy paste repeatedly into new projects.

As a result my larger projects have a bewildering mass of nodes that, several levels down, have many duplicate names.

Today I went through and renamed just the color related nodes one level down across about 30 level 1 subnetworks. This reduced the failure rate from 70% to 10%. If that 10% becomes a nuisance I can go one level deeper and rename more duplicate nodes.

This gets me back online and able to keep working. However, it took me the better part of an hour just to make these initial changes (testing periodically to ensure I didn't accidentally introduce new problems). So I now have a workaround, but a very tedious one.

I look forward to seeing this bug swatted for good in your next release.

Thanks for the update. It made my day!

John

Hi NodeBox team,

About a month ago you mentioned in another thread that one of my other bugs would be fixed in the next release, which would be coming soon.

Did you ever find a fix for the bug in this thread? Renaming everything is a workaround, but does become tedious for large projects.

Any new estimates on the date of the next release?

Thanks!

John

I've been trying to build a new release, but I have issues with the code signing certificate I use to verify the release is genuine (Otherwise Java applications will show a "this application is broken" message).

I've mailed Apple support for this, still waiting for a reply....

Any progress on this?

No, still waiting on Apple. I can create an unsigned build in the meantime, although I'm not a big fan of those...

Hello again.

Any progress on this? Surely Apple must have moved by now. Maybe if you sent them some Belgium chocolate...

Thanks,

John

I haven't heard back from Apple. I'll email them again.

I finally got a reply from Apple -- and they've given me new signing keys! So I'll try to get a new release out next week.

Best,

F

Hi Frederik,

Just downloaded and installed the new release on my Mac. Seems to be working fine. Thanks!

I had been under the impression that this version fixed the intermittent bug referenced in this thread (chance of crashes when multiple nodes in different subnetworks have the same name). But under the new version I am still seeing crashes with networks containing such nodes. Can you confirm whether or not version 3.0.44 attempted to fix this bug?

In a separate thread (Mar 11, 2015) I reported a problem with storkeWidth and strokeColor. Lookups on shapes with zero strokeWidth would return a strokeWidth of 1 and an undefined strokeColor. Stefan said this would be fixed in this version. The strokeWidth bug is indeed fixed, but the strokeColor bug remains. Is this correct?

Thanks again for continuing with updates.

John

The strokeWidth bug seems to be fixed in this release.

I just checked and the strokeColor doesn't seem to be fixed, though. I've filed this as a separate bug. Hopefully we'll be a bit faster with updates than before.

BTW you can always read the release notes to see what has changed. Or use the source and view the commit history.

I was just checking to be sure the release notes were 100% complete. To be clear, then, the intermittent bug referenced in this thread (chance of crashes when multiple nodes in different subnetworks have the same name) was verified but has NOT yet been fixed.

Is that bug still on your to do list? It still causes me trouble.

Thanks,

John

I've spent some time today looking at this bug. The issue is very difficult to track down, since the debug build doesn't give any errors. Even worse: if I just add print statements so I can sort-of see which line it happens on, the error goes away. A true Heisenbug.

The name of the node alone is of course not the issue. The whole Node instance is the key. Somehow, the Node instance is in there twice. This shouldn't occur, which is what the error is about. That's why changing the name makes sure that the error goes away: then they can't be the same instance anymore.

So, just a status update for now -- but I'll keep looking.

I like the name Heisenbug - but not the bug itself.

I just spent the last hour exhaustively hunting down every instance of subnetwork I often use to set colors according to a standard palette. Turns out there were six copies spread across scores of subsubsubsubsubnetworks. Had to rename a total of 60 nodes to make the bug go away, accidentally changing the rendered node as I went, causing more debugging.

(If we had true user-defined functions, as in NodeBox Live, I wouldn't need to keep copying subnetworks. Sure would be nice if you could bring that to NodeBox 3!)

If you can't swat this bug once and for all, I wonder if you could prevent it from happening in the first place by automatically renaming all nodes within a subnetwork whenever a user copies it. Not an ideal solution, but it shouldn't be too hard to do.

Please keep at it! Future generations will thank you.

Thanks,

John

I believe that user-defined functions are one of the best things in NodeBox Live, and I would love to port them over. Since that involves changes to the guts of NodeBox, maybe they would also fix the bug you're having.

However, at the moment I'm swamped with work for my PhD thesis. I'll ask one of my colleagues if they are willing to look at this bug again, for now.

I have worked on this issue, and it should work now. I've put an issue for this bug in the github repo here: https://github.com/nodebox/nodebox/issues/411

This can be added in for the next release.

Stefan,

Great news! Thanks for pursuing this nasty heisenbug to the bitter end.

I am attaching a simple network I just made to test the bug. It's a very simple pyramid of color subnetworks with several hundred duplicate nodes. Attempting to "Edit Children" of the root node reliably triggers the error.

I look forward to testing your fix in the next release (which I hope will be soon!).

Thanks again!

John