write_table Node

The attached demo contains a custom node, write_table, that turns any zip map into a csv file.

To use it, just attach any node that resolves to a table and double-click (render) the write_table node. If no errors occur the node will return "File Saved". You may then replace all the nodes used to create the table with a simple Import CSV node, open the file in Excel for further processing, or import it somewhere else (e.g. a separate NodeBox network).

The node takes two parameters, File Name and Delimiter. If you supply a simple file name (e.g. grocery list.csv) that file will appear on your desktop. To write to a specific folder, include that in the file path (e.g. groceries/feb2017.csv).

The delimiter is typically a comma but can be anything. If no delimiter is supplied, the node will use a tab (handy, since NodeBox can read tab-delimited files but will not let you enter a tab as a node parameter).

As with any custom node, you must copy/paste the node into your network, copy write_table.py into the same directory as your network, go to File/Code Libraries, use the + button in the upper right corner to add Python, and select write_table.py.

USES

This node is surprisingly useful, both for speeding performance of complex networks and for sharing information to separate Nodebox networks or elsewhere.

Complex NodeBox networks often have to import and process multiple csv or json files (sort them, merge them, filter them, etc.) before doing any drawing. If the files are large it may take ten or twenty seconds to repeat this process every time you change a color or add a new node somewhere, making design work tedious and greatly complicating the overall network.

With this node you can create one network to distill all the information you need into a single table, save that table as a csv file, and then create a simpler and faster network to import that table back in and focus on just the drawing part.

You can also use write_node to create a concordance file that supplies the data behind any SVG, PDF, or PNG you produce. For bubble charts, org charts, etc. you can save a separate file with data corresponding to each element in the chart along with its X,Y location in case you later want to filter or highlight particular values. A text concordance could supply searchable text and locations to supplement the unsearchable text paths produced by NodeBox.

Although we normally think of NodeBox as a tool for creating art and visualizations, it is also a great tool for data prep work. You can create a workflow that merges multiple data sources, sorts them, filters them, etc. and then saves the end result as a distilled csv file. This workflow can then be used again and again.

The write_table node is free to use by the NodeBox community for any purpose. If you find it useful please reply to this post and let me know.

Enjoy!

John

Awesome! This has been in the bug tracker since 2014.

so useful! thanks John! wanted a way of just extracting data to use in d3, this is ace— be great if built into NodeBox

Thanks, Sundar! Glad you found it useful.

I agree that this should be built in to NodeBox ASAP. It would probably be better to implement it as a new export option (add CSV to SVG, PNG, and PDF) instead of a node. That way you could just render any node and export it as a CSV without having to create a temporary node. The export dialog should include the delimiter option -including the ability to create tab-delimited files,

I've scheduled to add this feature to NodeBox this week!

awesome! thanks Frederik, and thanks again John! this will allow NodeBox to fit into so many workflows

I've just uploaded a new version of NodeBox that can export CSVs! It also supports custom delimiters, and correctly escapes strings with quotes.

In addition, it also exports graphical data: if the data is a Path, it will export the bounding box, SVG path data string, and fill and stroke color. If the data is a list of Points, it will export the X/Y values.

The build is not public yet (still have to make a Windows release), but you can find the Mac version here:

https://secure.nodebox.net/downloads/NodeBox-3.0.47.zip

And a Windows build (I haven't tested this myself yet, it's generated by AppVeyor): https://ci.appveyor.com/api/projects/fdb/nodebox/artifacts/dist/windows/bundles/NodeBox-3.0.47.exe

Frederik,

Yay! This is very cool. I have downloaded the Mac version and so far it's working great. I'll let you know if I run into any trouble.

The graphical CSV export was a pleasant surprise. I was particularly pleased that it exposes the "d" path with all the little codes inside (M for move, C for curve, L for line, etc.). This might make it easier to locate and then isolate or count particular shapes (paths) within a chart. This is something I have sometimes needed to do with simple shapes (e.g. using path length to isolate all circles in an org chart), but could not do with complex shapes or within complex charts.

One quick thing I did to demonstrate this was to create a textpath ("hello"), export it as a CSV, import that CSV back in, do a lookup on d, then feed that path into a make_strings node with Z as the separator. The result was individual paths for each letter: one path for simple letters and two paths for letters with contours ('e' and 'o'). In theory, I might be able to use that information to turn text shapes back into text. Cool!

[Note: I wish there were nodes that could turn a path into a d-notation string and vice versa. I could have a lot of fun with that. Currently in NodeBox all I can do with a path is turn it into a list of points; the distinction between lines and curves is lost.]

Thanks again for adding this feature. I have been using my write_table node more and more and have found it surprisingly useful. Now I will no longer need to keep adding that node each time I need to create a CSV. Hooray!

John

The code to read and write paths to SVG path data is already in there. It won't be too difficult to expose them.The names parse_svg_path_data and to_svg_path_data don't exactly roll of the tongue, so if you have better name suggestions... :-)

At the moment, I'm trying to do something about the difficulty in making new NodeBox releases: it takes a long time to make a release, because I have to switch back and forth between Mac and Windows to make the builds (and do a lot of other stuff, like updating the site). I'm looking at using AppVeyor (for Windows) and (Travis)[https://docs.travis-ci.com/user/osx-ci-environment/] or CircleCI for Mac. Having the build process fully automated will make it much easier to release builds on a more frequent basis.

Frederik,

One small oddity I've noticed with Export CSV...

The columns end up in a different order. For example, I had a table inside Nodebox with four columns: Country, Group, Code, and ID. When I exported it, the resulting CSV file had Group, Country, ID, and Code.

The ordering seems random. It's not alphabetical, and not reversed.

This is not a show stopper, but is mildly inconvenient. I often want my columns in a certain order. I can fix this in Excel, but it's a nuisance.

One other request if you get a chance to improve this wonderful feature: I wish the delimiter would default to comma instead of semi-colon. Excel defaults to comma and gets confused by semi-colons. I keep forgetting to change the delimiter and have it do it over each time.

My suggestion for SVG path node names is "Make Path - Convert a string to a path using SVG path notation" and "Path String - Convert path to a string using SVG path notation". If you don't get a chance to add these anytime soon, would there be an easy way to make them as custom nodes?

I LOVE LOVE LOVE csv export and am already using it heavily.

Thanks,

John

You're right about the ordering. That's because we first loop through
the objects, figuring out all of the keys, and adding them to a HashSet.
The HashSet doesn't guarantee ordering:

https://stackoverflow.com/questions/9345651/ordering-of-elements-in-java-hashset

However, a LinkedHashSet will; so maybe that's a better data structure
for this.

About the semicolon / colon: I actually found that Excel would open CSVs
directly if they had semicolons, but not if they have commas. That's why
I changed it. However, I personally like commas better (it's also what
Google Docs returns) so I'll change it back.

About the nodes, I'm pretty sure you could make them as custom nodes
through Python. The path to string would probably use
SVGRenderer
and string to path would need to use the SVG parser (which is already
written in
Python).

thanks for the updates, working fine on Win7 but had to remove previous version, otherwise it would not start

I use nodebox to read from a csv file and write hundreds of files to disk -kodi media stubs- it's not the same, as those are xml files and need to change filenames so even export range won't work, but maybe a generic write node would be useful

wonder if you could check into having 7 pins for the concatenate node

@rioch: Hmm, removing the old version is not so good. I've now moved to an automatic build system; I will see if I can provide a test for that.

What you're describing is something I've seen in vvvv, where they work with the concept of "sinks": nodes that do not have a visual output, but still need to be rendered. That would require a bit of a change, but it's not impossible.

As for the concatenate node, I've just made a commit that fixes this. Expect this in the next release!

@john I've also changed the default to colons, and added a fix to retain key ordering. That will also be in the next release (next week or so; I'm trying to get build automation in order).

Frederik,

I assume when you say "changed the default to colons" you meant commas. Right?

It's exciting to see these improvements.

John

thanks for the 7 ports then, it usually takes one more concatenate nodes than it should
about that writer node it would allow exporting formats like xml css obj and many more... nice you mentioned vvvv, it's cool that v4 spreads and nb3 lists work in similar ways, but they are also trying new things with VL

Frederik,

I just ran into an issue with the new CSV export. I used it to export a simple list in dot notation (so I could draw it using Graphviz).

My write node works fine for this. But when I tried to use Export CSV instead, each row of the list was enclosed in quotes. I guess this is the "correctly escapes strings with quotes" feature you referred to above.

In this case, adding quotes made the file unreadable by Graphviz. I can imagine many other situations in which adding quotes would be disastrous.

So can you make some way to turn this "feature" off?

Thanks!

John

I'm not sure how Graphviz interprets the data. I can understand that it wouldn't like quotes around numeric data. I can see if I can keep them around text but remove them around numbers (or make it an option, as you say).

Frederik,

The lines in the file look like this:

digraph Flow {
node [shape=box] ;
FndOverviewTF -> AddPwkMainFlow ;
AddPwkMainFlow -> AddPwkMainFlow ;

No numbers, but spaces and lots of punctuation. So it's text, not numbers, but I still don't want quotes. Maybe it should just be an option.

I can do that. Just be aware that anything that will include quotes will not work (since the quotes are the only escape mechanism CSV has).

FYI here's the GitHub issue tracking this: https://github.com/nodebox/nodebox/issues/434

Thanks, Frederik. I understand that if I am exporting files with special formatting requirements I am responsible for doing that formatting myself.

One followup.

What I'm really doing in this case is creating a text file from a list. This sounds like what rioch is doing with his XML files.

Because this is a list, not a multi-column table, I don't need to worry about commas inside strings being misinterpreted as delimiters. I just want the actual (unmodified) text.

I suppose you could add a separate ability to export lists as text files. This would save me from constructing a one-column table with the first line of text as the column header, and from removing the ".csv" after export. But CSV export is simple and works on all nodes. I can live with that.